Deep Learning

An artificial neuron is a simple mathematical unit loosely inspired by biological neurons: it takes a set of numeric inputs, multiplies each by a learned weight, sums the results together with a bias term, and passes that sum through a nonlinear activation function to produce an output. Early neurons used step functions or the sigmoid function, which squashes any input into a smooth range between zero and one, but modern networks overwhelmingly favor the rectified linear unit, or ReLU, which simply outputs the input value when it is positive and zero otherwise. This nonlinearity matters enormously: without it, stacking layers of neurons would collapse mathematically into a single linear transformation no more expressive than one layer, regardless of how many layers were stacked. A single neuron together with an activation function is often called a perceptron, and a layer of many such neurons operating on the same inputs is called a fully connected or dense layer. When a network is arranged so that its output feeds forward through one or more layers of neurons that sit between the input and the output, those intermediate layers are called hidden layers because their values are not directly observed in the training data.

A network is conventionally described as deep when it contains multiple hidden layers stacked between input and output, as opposed to a shallow network with only one. Depth gives a network the ability to build up hierarchical representations, where early layers might respond to simple patterns and later layers combine those patterns into increasingly abstract features, though the precise nature of the representations depends heavily on the task and data. This layered composition is what gives deep learning its name and its practical power, since a sufficiently deep and wide network can, in principle, approximate extremely complex functions that a shallow network could only approximate poorly or not at all. Stanford's CS231n course notes describe this compositional structure carefully, framing a neural network as a stack of differentiable functions chained together, where each layer's output becomes the next layer's input. The number of layers, the number of neurons per layer, and the choice of activation function are all architectural decisions that a practitioner must make before training even begins, and they collectively define what is often called the network's capacity.

Training a deep network means finding the weight and bias values that make its predictions match the desired outputs as closely as possible, and the algorithm that makes this tractable is backpropagation, formalized in the influential 1986 Nature paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams titled 'Learning representations by back-propagating errors.' Training proceeds in repeated cycles: first, a forward pass sends a batch of input data through the network layer by layer to produce a prediction; second, a loss function compares that prediction against the true target and computes a single number summarizing how wrong the network was. The backward pass then applies the chain rule of calculus to propagate that error signal backward through the network, computing the gradient, or the rate of change of the loss, with respect to every single weight and bias in every layer, starting from the output layer and working back toward the input. Rumelhart, Hinton, and Williams showed that this gradient computation could be organized efficiently by reusing intermediate results from later layers when computing gradients for earlier layers, which is precisely why the method scales to networks with many layers. Once every gradient is known, an optimization step called gradient descent nudges each weight a small amount in the direction that reduces the loss, and repeating this forward-loss-backward-update cycle over many examples gradually shapes the network into one that performs the task well.

Fully connected networks treat every input independently and ignore spatial structure, which becomes wasteful and impractical for images, where nearby pixels are highly correlated and a useful pattern such as an edge or a texture can appear anywhere in the frame. Convolutional neural networks address this by replacing dense connections with convolutional filters, small learnable grids of weights, typically a few pixels wide and tall, that slide across the width and height of an image and compute a dot product at every position. Because the same filter is reused at every spatial location, the network learns one compact pattern detector, such as a vertical edge detector or a color-contrast detector, and applies it uniformly across the whole image rather than learning a separate detector for every pixel position, which drastically reduces the number of parameters compared to a fully connected layer of similar size. The output of sliding one filter across an image is called a feature map, and a convolutional layer typically learns many filters in parallel, producing a stack of feature maps that each highlight a different low-level pattern. As described in Stanford's CS231n course notes, stacking several convolutional layers allows the network to build increasingly complex and abstract feature detectors, with early layers responding to edges and colors and deeper layers responding to textures, parts, and eventually whole object categories.

Convolutional layers are typically interleaved with pooling layers, which downsample feature maps by summarizing small neighborhoods, most commonly by taking the maximum activation within each local region in an operation called max pooling. Pooling serves two purposes: it reduces the spatial size of the data flowing through the network, which cuts computational cost and memory use in later layers, and it introduces a degree of translation invariance, meaning the network's response becomes less sensitive to small shifts in exactly where a pattern appears in the image. A typical CNN architecture alternates convolutional layers, activation functions, and pooling layers several times, progressively shrinking the spatial dimensions of the feature maps while increasing their depth, before finally flattening the result into one or more fully connected layers that produce the final classification output. This combination of local connectivity, weight sharing across spatial positions, and pooling is what makes convolutional networks dramatically more parameter-efficient and better suited to image data than naively applying a fully connected network to raw pixels.

The practical power of deep convolutional networks was demonstrated dramatically by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton in their 2012 paper 'ImageNet Classification with Deep Convolutional Neural Networks,' describing the architecture that came to be known as AlexNet. Trained on the ImageNet Large Scale Visual Recognition Challenge dataset spanning roughly 1.2 million labeled images across 1,000 categories, AlexNet was an eight-layer network combining five convolutional layers with three fully connected layers, trained using ReLU activations and dropout regularization on graphics processing units. On the ILSVRC-2010 test set, the paper reports that the network achieved top-1 and top-5 error rates of 37.5% and 17.0% respectively, and an ensemble variant of the architecture went on to win the ILSVRC-2012 competition with a top-5 error rate of 15.3%, far outperforming the next best entry, which relied on traditional hand-engineered computer vision features rather than learned representations. This result is widely credited with convincing the broader computer vision community that deep convolutional networks, trained end to end on large labeled datasets with sufficient GPU compute, could outperform decades of hand-crafted feature engineering, and it is commonly cited as the catalyst for the deep learning boom of the following decade.