Convolutional Neural Networks

A convolutional layer's output spatial size is determined by four quantities: the input size W, the filter size F, the stride S with which the filter slides across the input, and the amount of zero-padding P added around the input's border, related by the formula (W − F + 2P) / S + 1. A concrete example makes this tangible: sliding a 5x5 filter with stride 1 and no padding across a 32x32 input produces a (32 − 5 + 0) / 1 + 1 = 28x28 output. This is not an arbitrary example; it is precisely the transformation performed by the first convolutional layer of LeNet-5, the architecture Yann LeCun and colleagues described in their 1998 paper "Gradient-Based Learning Applied to Document Recognition," published in the Proceedings of the IEEE. Padding is used when a layer needs to preserve its input's spatial size rather than shrink it, while a stride greater than one is itself an alternative way to downsample a feature map, distinct from using a separate pooling layer for the same purpose.

As a network gets deeper, each unit's receptive field, meaning the region of the original input that can influence that unit's value, grows layer by layer. Stanford's CS231n course notes work through the case of stacking three 3x3 convolutional layers: a neuron in the second layer already has an effective 3x3 view of the first layer's output, and by extension a 5x5 view of the original input, while a neuron in the third layer has a 3x3 view of the second layer and therefore a 7x7 view of the input, matching the receptive field of a single 7x7 convolutional layer applied directly. The notes point out that this stacked arrangement uses far fewer parameters than the single larger filter: for C input and output channels, three 3x3 layers together use 3 × (3 × 3 × C × C) = 27C² weights, compared to 7 × 7 × C × C = 49C² weights for one 7x7 layer, while also inserting two additional nonlinearities between the smaller layers that make the resulting composed function strictly more expressive than the single large filter's purely linear combination.

LeNet-5's own architecture, built for recognizing handwritten and machine-printed characters, alternated two convolutional layers using 5x5 kernels and sigmoid nonlinearities with two average-pooling, or subsampling, layers, before feeding into three fully connected layers with 120, 84, and 10 units, the last producing a score for each of the ten digit classes. Average pooling, rather than the max pooling used by most later architectures, was the downsampling choice available to LeCun and colleagues at the time, and it served the same basic purpose of shrinking the spatial size of the feature maps flowing through the network while retaining a summary of each local region's activity. The architecture achieved well under 1% error on digit-recognition benchmarks of the era and was deployed commercially to read handwritten amounts on bank checks, standing as an early, working proof that a trainable, backpropagation-based convolutional architecture could operate reliably at industrial scale, more than a decade before an architecture built on the same core convolutional principles, described in an earlier module of this course, went on to transform image classification on far larger, natural-image datasets.

A convolutional layer's central assumption is that images have translationally structured content: if a small filter is useful for detecting some pattern, such as an oriented edge, at one spatial position, that same filter is likely to be useful for detecting the same pattern at a different position elsewhere in the image. CS231n's course notes frame the resulting design choice, weight sharing, as a direct consequence of this assumption: rather than learning an independent weight for every pixel position, a convolutional layer reuses one compact set of filter weights across the entire spatial extent of its input, which both drastically reduces the layer's parameter count relative to a fully connected layer of comparable size and enforces local connectivity, since each unit's output depends only on a small spatially local patch of the layer's input rather than the input as a whole. This same reasoning is what makes convolution a poor fit for data without that spatial regularity, since the entire efficiency argument rests on a pattern's usefulness genuinely being independent of where it happens to sit in the frame.

Because each layer's receptive field is built by composing the receptive fields of the layer beneath it, stacking convolutional layers lets a network build up a hierarchy of increasingly abstract features without ever being told explicitly what to detect at each layer. Early layers, operating on small receptive fields close to the raw pixels, tend to develop into generic low-level detectors for edges and color contrasts; deeper layers, whose receptive fields span much larger regions of the original input, can combine those low-level responses into detectors for textures, object parts, and eventually whole categories. This layered composition emerges purely from training a stack of convolution, nonlinearity, and pooling operations end to end with backpropagation on labeled data, with no hierarchy imposed by hand; the same general filter-stacking mechanics described in this module's first lesson, applied repeatedly across many more layers than a shallow network would use, are what let this abstraction climb from raw pixel intensities all the way up to object-level concepts.

It is worth distinguishing two related but different properties that this architecture produces. Weight sharing gives a convolutional layer translation equivariance: shifting the input by some amount shifts the resulting feature map's response by a corresponding amount, preserving spatial position. Pooling layers, by contrast, are what introduce a degree of translation invariance, deliberately discarding some precise spatial information by summarizing small neighborhoods, so that the network's ultimate classification becomes less sensitive to exactly where within the frame a pattern appears. This creates a real trade-off: aggressive pooling throughout a network is well suited to whole-image classification, where only the presence of a pattern matters, but tasks that require knowing precisely where a pattern is located need the network to retain more of the spatial detail that pooling discards, which is why classification-oriented architectures and localization-oriented architectures tend to make different choices about how much and how often to pool.

By the mid-2010s, researchers observed a puzzling phenomenon in very deep, plain convolutional networks, meaning ordinary stacks of convolutional layers with no special connectivity beyond one layer feeding the next. Naive reasoning suggests that adding more layers to a working network should never make it worse: in the worst case, the additional layers could simply learn to pass their input through unchanged, reproducing the shallower network's performance exactly. Kaiming He and colleagues' 2015 paper on deep residual learning showed that, empirically, this is not what happens; past a certain depth, plain networks' training error itself, not just their test error, gets measurably worse as more layers are added, which rules out overfitting as the explanation, since overfitting would show low training error alongside high test error rather than both rising together. He et al. term this the degradation problem, an optimization difficulty distinct from the vanishing- or exploding-gradient issues that careful initialization and normalization techniques had already substantially addressed.

The paper's proposed fix reframes what each block of layers is asked to learn. Instead of requiring a stack of layers to learn some desired underlying mapping H(x) directly from scratch, a residual block is restructured to learn a residual function F(x) = H(x) − x, with the block's actual output computed as F(x) + x, implemented by adding a shortcut connection that carries the block's input x forward unchanged and sums it with the block's ordinary output. He et al.'s reasoning is that if the optimal mapping for a given block happens to be close to the identity function, it is far easier for gradient-based optimization to push a residual function F(x) toward zero than it is for a stack of nonlinear layers to learn to approximate an identity mapping directly, and this reframing is what let their architectures train successfully at depths where plain networks could not. Critically, the shortcut connection adds no extra parameters and no meaningful extra computation to the network, since it is simply an addition operation performed on values the network was already computing, so the improvement comes entirely from restructuring what the existing weight layers are asked to represent rather than from any added model capacity.

The empirical results were substantial: the authors built residual networks up to 152 layers deep, described as eight times deeper than a comparable VGG network while carrying lower computational complexity, and separately trained plain networks of over 1,000 layers on CIFAR-10 to study the effect directly. An ensemble of their residual networks achieved 3.57% top-5 error on the ImageNet test set, taking first place in image classification at the ILSVRC 2015 competition, and residual-network-based models also placed first that year in ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation, indicating the residual-learning idea generalized well beyond image classification alone. The shortcut connection introduced to solve one specific degradation problem went on to become a standard architectural building block adopted across most very deep network designs that followed it, extending well beyond convolutional networks for images into other domains that stack many learned layers.