3 ResNet Revolution
Deep neural networks once faced a frustrating trade-off: greater depth promised richer representations, but adding layers often made optimization worse rather than better. Models suffered from degraded forward signals and vanishing gradients, so deeper “plain” networks could produce higher training error than shallower ones. Residual networks changed this by introducing shortcut connections that let information and gradients flow around groups of layers. Instead of forcing every layer to learn a full transformation, residual blocks learned small refinements relative to the input, making it safe for layers to behave like identity mappings when no useful transformation was needed.
This simple design produced a major leap in computer vision. ResNet made very deep models practical, efficient, and highly accurate, outperforming earlier architectures such as AlexNet, VGGNet, and GoogLeNet while scaling to variants with dozens, hundreds, and even more than a thousand layers. Its bottleneck blocks reduced computation by compressing channels before expensive convolutions and expanding them afterward, helping models like ResNet-152 achieve strong ImageNet results with fewer operations than much shallower alternatives. ResNet v2 refined the idea further by moving normalization and activation before convolutions, preserving a cleaner identity path and enabling more stable optimization in ultra-deep networks.
The chapter also traces ResNet’s broader influence. Residual connections became a default building block across computer vision and later deep learning architectures, appearing in production systems, detection and segmentation models, medical imaging, autonomous driving, and hybrid designs. For dense prediction tasks such as semantic segmentation, the chapter highlights dilated convolutions, which preserve spatial detail while expanding receptive fields without pooling or excessive parameters. It also critiques the public celebration of “human-level” ImageNet performance, arguing that benchmark success did not imply human-like understanding or robustness. Finally, it notes how Stanford’s CS231n helped spread these breakthroughs globally, turning the architecture-driven deep learning revolution into a widely taught and practiced discipline.
Diagram of a simple ResNet block. It illustrates how input data flows along two paths: one through two transformation layers with activation steps, and another around via a direct “shortcut” path with no transformations. The outputs of these two paths are combined, and the resulting vector undergoes a final activation step. This “shortcut” helps the network learn more effectively by preserving information from earlier layers. Used with explicit permission granted by the lead author (Kaiming He).
Training error (left) and test error (right) on CIFAR-10 with 20-layer and 56-layer “plain” networks. The deeper 56-layer network shows higher training error than the shallower 20-layer network, indicating that it is harder to optimize without additional architectural innovations. It also has a higher test error (right), showing that simply adding layers doesn’t improve performance. In both plots, the x-axis denotes training iterations in units of 10,000; each iteration corresponds to a single mini-batch update. As training progresses across approximately 60,000 iterations, the deeper network consistently underperforms. Used with permission granted by the lead author (Kaiming He).
Thin error curves indicate training error on ImageNet, and bold error curves indicate validation error on ImageNet. Left: plain networks with 18 and 34 layers. Right: ResNets with 18 and 34 layers. Residual networks have no additional parameters than their plain counterparts. Used with explicit permission granted by the lead author (Kaiming He).
Left: the basic block used in ResNet-34 passes a 64-channel input through two 3×3 convolutions and then adds the original input back before applying ReLU. Right: the bottleneck block used in deeper ResNets first compresses features with a 1×1 convolution, processes them with a 3×3 layer, and then restores dimensionality with another 1×1 convolution. The shortcut bypass preserves the original signal, ensuring that deeper models refine features rather than lose them. Used with explicit permission granted by the lead author (Kaiming He).
Left: the original residual unit shows two “weight” layers (i.e., convolutional layers), with batch normalization (BN) and ReLU activations between them, followed by an additional step that merges back the identity connection. Right: the revised residual unit reorganizes the operations by placing BN and ReLU before the weight layers, simplifying the pathway. The gray arrows highlight the direct paths (i.e., skip connections) that facilitate the smooth forward and backward flow of information through the residual (shortcut) connections. Used with explicit permission granted by the lead author (Kaiming He).
Dilated convolutions rapidly expand the receptive field without increasing computational cost. Each red dot represents a convolutional kernel element. As dilation increases (from left to right), the receptive field (blue-shaded area) expands exponentially, capturing broader spatial context while keeping the parameter count the same. Used with explicit permission granted by the lead author (Fisher Yu).
Semantic segmentations produced by different classification networks. From left to right: the first column is the input image, the second is the prediction by FCN-8s, the third is the prediction by DeepLab, the fourth is the prediction by Yu and Koltun (2015), and the last is the ground truth. The examples illustrate progressively more precise and accurate segmentations, with the method of Yu and Koltun (2015) significantly improving boundary definition and object identification compared to previous methods. Used with explicit permission granted by the lead author (Fisher Yu).
Scale What?
ResNet emerged as a pragmatic response to model degradation. As depth increased, networks performed worse. Residual connections and, later, pre-activation restored signal flow. That stability created a flywheel effect: small architectural tweaks unlocked larger models, and larger models increased the payoff of the architecture.
This flywheel is specific to deep learning.
By contrast, a Support Vector Machine is a shallow margin maximizer. It selects a decision boundary in a fixed feature space (explicit or kernel-defined). There are no stacked representations or gradients to propagate through depth, so there’s no analog of vanishing gradients or model degradation as networks get deeper. In fact, there’s no “deeper” knob to turn. While you can use more compute to solve the convex optimization more precisely, that doesn’t create the deep-learning-style scaling loop. So this architecture renders innovations like residual connections irrelevant.
In contrast, computation and data compound only when the architecture reliably converts them into richer learned representations. That’s precisely the regime that residual connections enable in deep learning. Residual connections matter only when a model iteratively learns layered representations through gradient-based optimization across many layers.
But do they?
Data and compute are necessary but insufficient. The decisive question is not whether we have more of them, but whether the architecture and training paradigms can use them effectively. One reason a reading list like Sutskever’s matters is that it makes legible the inventions that made scaling pay off in the first place and the people behind them. If Stone Soup and The Bitter Lesson tell researchers to scale, Ilya’s rejoinder poses the more actionable research question: scale what?
In 2012, AlexNet ignited the deep learning revolution, catapulting the ILSVRC into the defining battlefield for computer vision. Yet barely half a decade later (and just two years after ResNet’s introduction), ImageNet reached a saturation point. The contest, regarded by Jitendra Malik as ahead of its time, had been solved, prompting Fei-Fei Li to discontinue the competition. Researchers shifted their gaze toward new frontiers. In the aftermath, the ImageNet dataset transitioned from a cutting-edge benchmark to just another baseline. It now occupies a more symbolic position in computer vision research, akin to MNIST, a dataset of handwritten digits that has long served as an introductory learning tool and basic reference point for beginners, rather than a source of ongoing innovation.
By 2025, the original ResNet paper had amassed over 100,000 citations in the academic literature and roughly 250,000 on Google Scholar, making it the most-cited scientific paper of the 21st century across all fields.[51] Virtually every new computer vision paper referenced, compared against, or built upon ResNet, solidifying its role as the standard architecture in the field. Residual connections also transcended computer vision, fueling breakthroughs in domains ranging from AlphaGo’s mastery of board games and AlphaFold’s protein-structure predictions to advances in language modeling like GPT-2. By enabling deep networks, residual connections have become essential to modern artificial intelligence across various domains.
Yet computer vision was only one front in the broader deep learning revolution. As models expanded into sequences, speech, and language, they also grew larger, more complex, and more demanding. This scaling placed new pressure on the field, shifting progress away from isolated architectural insights and toward the engineering discipline required to train, optimize, and deploy systems at scale. That shift began with the practical triumph of Deep Speech 2, where performance often emerged from the careful integration of ideas, infrastructure, and computation.
FAQ
Why were very deep neural networks difficult to train before ResNet?
Before ResNet, simply adding more layers often made neural networks harder to optimize. Deeper “plain” networks suffered from model degradation, where forward signals became increasingly distorted, and from vanishing gradients, where backward learning signals became too weak to update early layers effectively. As a result, deeper models could have higher training error than shallower ones, even though they should theoretically have greater representational capacity.
What is the main idea behind residual connections?
Residual connections create shortcut paths that allow information to flow around groups of layers instead of being forced through every transformation sequentially. A residual block learns an incremental change, or residual, and adds it to the original input:
y = F(x) + x
Here, F(x) is the transformation learned by the block, while x is the unchanged input. This helps preserve useful forward signals and provides clearer gradient pathways during backpropagation.
How did ResNet improve performance at ILSVRC 2015?
ResNet produced a major jump in ImageNet performance. At ILSVRC 2015, an ensemble of ResNets achieved a 3.57% top-5 error rate, while a single ResNet achieved 4.49%. ResNet won across classification, localization, and detection categories, roughly halving the previous year’s top-5 error rate in a way comparable to AlexNet’s earlier breakthrough.
Why does ResNet make it safer to add more layers?
Residual connections allow layers to perform identity mapping: if a layer does not learn a useful transformation, it can effectively pass its input forward unchanged. This means extra layers do not necessarily damage performance. Instead of forcing every layer to transform the data, ResNet lets layers learn only useful residual adjustments relative to the input.
What is a bottleneck block in ResNet?
A bottleneck block is an efficient residual block design used in deeper ResNets. It first uses a 1×1 convolution to reduce the number of channels, then applies a 3×3 convolution in this lower-dimensional space, and finally uses another 1×1 convolution to expand the representation again. This reduces computational cost while preserving expressive power, helping models like ResNet-152 remain efficient despite their depth.
How did ResNet compare with VGGNet and GoogLeNet?
VGGNet used repeated 3×3 convolutional layers and achieved strong results, but it had a very large parameter count and became difficult to train beyond about 19 layers. GoogLeNet used manually engineered Inception modules with multiple parallel paths, achieving efficiency but requiring careful tuning. ResNet’s residual block was simpler and more scalable, enabling architectures such as ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152, and even ResNet-1202 without redesigning the core idea.
What changed in ResNet v2?
ResNet v2 introduced pre-activation residual blocks. In the original ResNet, a ReLU activation was applied after adding the residual path and skip connection. ResNet v2 removed this post-addition activation and moved batch normalization and ReLU before the convolutional layers. This preserved the skip connection as a clean identity path, improving gradient flow and making extremely deep networks easier to optimize.
What did the ResNet v2 ablation studies show?
The ResNet v2 authors tested modifications to the skip connection, such as scaling it, randomly dropping it, adding gates, or inserting convolutional layers. These changes generally worsened performance or caused optimization problems. The studies showed that the simplest unaltered identity skip connection worked best, especially in very deep networks where small architectural changes could make training succeed or fail.
Why are dilated convolutions important for dense prediction tasks?
Dense prediction tasks, such as semantic segmentation, require a label for every pixel and therefore need high spatial precision. Pooling and strided convolutions help classification models but discard fine detail. Dilated convolutions expand the receptive field by inserting gaps between kernel elements, allowing models to capture wider context without reducing resolution or increasing the number of parameters per layer. This made them valuable for segmentation and other dense prediction problems.
Why is Stanford CS231n included in this chapter?
CS231n was Stanford’s first full-length course devoted entirely to deep learning and reflected the major vision breakthroughs of the era, including AlexNet, VGGNet, GoogLeNet, and ResNet. Created by Fei-Fei Li, Andrej Karpathy, and Justin Johnson, the course made lectures, notes, slides, and assignments freely available online. It helped spread deep learning knowledge globally and became a major educational force in computer vision.
Sutskever's List ebook for free
