A residual neural network (also referred to as a residual network or ResNet) is a deep learning architecture in which the layers learn residual functions with reference to the layer inputs. It was developed in 2015 for image recognition, and won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) of that year.

As a point of terminology, "residual connection" refers to the specific architectural motif of ${\displaystyle x\mapsto f(x)+x}$, where ${\displaystyle f}$ is an arbitrary neural network module. The motif had been used previously (see §History for details). However, the publication of ResNet made it widely popular for feedforward networks, appearing in neural networks that are seemingly unrelated to ResNet.

The residual connection stabilizes the training and convergence of deep neural networks with hundreds of layers, and is a common motif in deep neural networks, such as transformer models (e.g., BERT, and GPT models such as ChatGPT), the AlphaGo Zero system, the AlphaStar system, and the AlphaFold system.

In a multilayer neural network model, consider a subnetwork with a certain number of stacked layers (e.g., 2 or 3). Denote the underlying function performed by this subnetwork as ${\displaystyle H(x)}$, where ${\displaystyle x}$ is the input to the subnetwork. Residual learning re-parameterizes this subnetwork and lets the parameter layers represent a "residual function" ${\displaystyle F(x)=H(x)-x}$. The output ${\displaystyle y}$ of this subnetwork is then represented as:

$${\displaystyle y=F(x)+x}$$

The operation of "${\displaystyle +\ x}$" is implemented via a "skip connection" that performs an identity mapping to connect the input of the subnetwork with its output. This connection is referred to as a "residual connection" in later work. The function ${\displaystyle F(x)}$ is often represented by matrix multiplication interlaced with activation functions and normalization operations (e.g., batch normalization or layer normalization). As a whole, one of these subnetworks is referred to as a "residual block". A deep residual network is constructed by simply stacking these blocks.

Long short-term memory (LSTM) has a memory mechanism that serves as a residual connection. In an LSTM without a forget gate, an input ${\displaystyle x_{t}}$ is processed by a function ${\displaystyle F}$ and added to a memory cell ${\displaystyle c_{t}}$, resulting in ${\displaystyle c_{t+1}=c_{t}+F(x_{t})}$. An LSTM with a forget gate essentially functions as a highway network.

To stabilize the variance of the layers' inputs, it is recommended to replace the residual connections ${\displaystyle x+f(x)}$ with ${\displaystyle x/L+f(x)}$, where ${\displaystyle L}$ is the total number of residual layers.