Artificial Intelligence
Deep Residual Learning for Image Recognition
He, Zhang, Ren, Sun · 2015 · CVPR
Residual connections enabled training of very deep networks, winning ImageNet 2015.
Research objective
Solve the degradation problem in which adding layers to deep networks paradoxically harms training accuracy.
Methodology
Introduced 'identity shortcut' connections that let layers learn residual functions F(x) + x. Trained networks of 50, 101, and 152 layers on ImageNet.
Key findings
- 152-layer ResNet won ILSVRC 2015 with 3.57% top-5 error, surpassing human-level performance on the benchmark.
- Residual connections dramatically eased optimization of deep networks.
- Generalized to detection, segmentation, and many downstream tasks.
Strengths
- Tiny architectural change with outsized impact.
- Made very deep networks practically trainable.
- Became a default building block across computer vision and beyond.
Limitations
- Did not eliminate vanishing gradients entirely; depth still has diminishing returns.
- Convolutions later partially supplanted by Vision Transformers in some regimes.
Practical implications
- Residual connections appear in nearly every modern deep architecture, including Transformers.
- Established depth-via-skip-connections as a foundational design pattern.
Related entities
Related research
Attention Is All You Need
Introduced the Transformer, replacing recurrence with self-attention and reshaping modern AI.
Read summary
Mastering the Game of Go with Deep Neural Networks and Tree Search
AlphaGo combined deep networks and Monte Carlo tree search to defeat world-class Go players.
Read summary