Deep-Network Training and Optimization

Before a single gradient step is taken, the values a network's weights are initialized to already determine whether training can succeed at all. Setting every weight to zero is the clearest failure case: every neuron in a layer then computes an identical output from any given input, receives an identical gradient during backpropagation, and undergoes an identical update, so the neurons stay tied together indefinitely no matter how long training continues, and the layer never develops the differentiated feature detectors it needs. Small random values break this symmetry, but the scale of that randomness turns out to matter enormously. Xavier Glorot and Yoshua Bengio's 2010 analysis of deep feedforward networks showed that with a commonly used naive initialization scheme, sigmoid activations in the last hidden layer get pushed toward their saturated extremes early in training, and, more generally, the variance of the back-propagated gradient shrinks as it moves from the output layer back toward earlier layers, so gradients can effectively vanish before they ever reach a network's first few layers.

Glorot and Bengio proposed a normalized initialization designed to keep the variance of activations roughly constant on the forward pass and the variance of gradients roughly constant on the backward pass, even across layers of different widths, arriving at weights drawn from a distribution scaled by the sum of a layer's number of input and output units. Their experiments showed this normalized scheme substantially reduced test error across several benchmarks relative to standard initialization, for example lowering error on their Shapeset-3x2 task from 27.15% to 15.60% with tanh networks. Because rectified linear units behave differently from sigmoid or tanh units, a later refinement scales the initialization variance using only the number of input units rather than the sum of inputs and outputs, drawing weights with standard deviation proportional to the square root of two divided by that input count; the extra factor of two accounts for the fact that a ReLU unit zeroes out roughly half of its activations by construction, so the variance carried forward through the remaining active half must be compensated for to keep signal strength from shrinking layer by layer. CS231n's course notes describe this ReLU-specific scaling as the current recommendation for use in practice.

Even with careful initialization, the distribution of each layer's inputs keeps shifting during training as the parameters of every preceding layer change, a phenomenon Sergey Ioffe and Christian Szegedy term internal covariate shift, and which slows training by forcing later layers to continuously readjust to a moving target. Their 2015 batch normalization technique addresses this directly by normalizing each layer's inputs, computed over each training mini-batch, to have roughly zero mean and unit variance before applying a learnable scale and shift, typically inserting this normalization step between a layer's linear transformation and its nonlinearity. Ioffe and Szegedy report that batch-normalized networks reach the same accuracy as an unnormalized baseline using 14 times fewer training steps, tolerate substantially higher learning rates, and become markedly less sensitive to the initialization scheme in the first place; an ensemble of their batch-normalized networks reached 4.9% top-5 error on ImageNet validation data, exceeding the accuracy of human raters reported in the same line of work.

Regularization techniques exist to keep a network from fitting the idiosyncrasies of its training set so tightly that it fails to generalize to new data. The most direct approach, L2 regularization or weight decay, adds a penalty term proportional to the squared magnitude of each weight to the loss being optimized; because gradient descent then includes a term that shrinks every weight slightly toward zero on every update, no single weight is free to grow arbitrarily large and dominate the network's output, which tends to push the network toward spreading its representational work across many weights rather than concentrating it in a few. CS231n's course notes describe this update as decaying every weight linearly toward zero on each step, in addition to whatever movement the loss's own gradient calls for, which is exactly why the technique is commonly called weight decay rather than simply a loss penalty: its practical effect is visible directly in the parameter-update rule itself, not just in the value of the loss function being minimized.

Nitish Srivastava and colleagues' 2014 dropout technique takes a structurally different approach: during training, each unit, together with all of its incoming and outgoing connections, is temporarily and randomly removed from the network with some fixed probability, commonly one-half for hidden units, and a fresh random subset of units is dropped on every forward and backward pass. The stated purpose, in the authors' own framing, is to prevent units from co-adapting too much, since any given unit cannot rely on any particular other unit being present on a given pass and so must learn features that are useful in combination with many different, randomly chosen subsets of the rest of the network. Because a new random subnetwork is effectively sampled on every training step, Srivastava et al. describe training under dropout as approximately training an exponential number of distinct thinned networks that share weights, an implicit ensemble that no single ordinary training run could construct explicitly.

At test time, dropout is turned off and predictions come from the single full, unthinned network, which is used as an efficient approximation to averaging the predictions of that exponential ensemble of thinned networks; the common inverted-dropout implementation divides each retained activation by the same keep probability during training itself, so no separate rescaling step is needed at test time. Srivastava et al. report that dropout improved performance across supervised tasks spanning vision, speech recognition, document classification, and computational biology, and produced state-of-the-art results on several benchmark datasets at the time of publication. Weight decay and dropout address overfitting through different mechanisms, one constraining weight magnitude directly and the other perturbing the network's effective architecture on every step, and the two are frequently used together rather than as alternatives. A useful way to compare the two techniques is by their computational cost relative to what they approximate: literally training and averaging predictions from an exponential number of separate networks would be intractable, but dropout obtains a similar regularizing effect at the cost of one extra random mask per forward pass, making the ensemble-like benefit essentially free compared to the alternative of training many independent models and combining their outputs by hand.

Plain stochastic gradient descent updates each parameter by moving it a small step opposite its gradient, following the simple rule that a parameter x is updated as x plus negative learning rate times the gradient, which is guaranteed to improve the loss whenever the learning rate is small enough, but this simple rule is easily thrown off by noisy or inconsistent gradient directions from one mini-batch to the next. Momentum reframes the update as a physical particle rolling downhill: a velocity term accumulates a running combination of past gradients, typically decayed by a factor between 0.9 and 0.99 on each step, so the parameter builds up speed in directions where the gradient consistently points the same way and has its motion damped in directions where the gradient keeps flipping sign. Nesterov momentum refines this further by evaluating the gradient not at the current position but at the position the momentum term is already about to carry the parameters toward, a look-ahead correction that CS231n's course notes describe as giving stronger theoretical convergence guarantees for convex objectives than standard momentum alone.

A separate family of methods adapts the effective learning rate per parameter rather than per training step. Adagrad divides each parameter's update by the square root of that parameter's accumulated sum of squared past gradients, which helps rarely updated parameters take larger steps, but because that accumulated cache only grows, the effective learning rate keeps shrinking and, per the course notes, often stops learning too early in deep networks. RMSProp replaces the ever-growing sum with an exponentially decaying moving average of squared gradients, preventing the monotonic decay problem. Diederik Kingma and Jimmy Ba's 2014 Adam optimizer combines both ideas at once: it maintains an exponentially decaying average of past gradients themselves as a first-moment estimate, functioning like momentum, and a separate exponentially decaying average of past squared gradients as a second-moment estimate, functioning like RMSProp's adaptive scale, by default decaying each at rates of 0.9 and 0.999 respectively, then updates parameters using the first-moment estimate divided by the square root of the second-moment estimate plus a small constant for numerical stability. The authors describe Adam as computationally efficient, requiring little memory, well suited to problems with noisy or sparse gradients, and, per CS231n's notes, it is currently recommended as the default optimizer to reach for in practice.

Choosing hyperparameters and diagnosing a training run both rely on reading the shape of a few key signals rather than guessing. A loss curve that improves roughly linearly suggests too low a learning rate, while one that oscillates or trends upward suggests too high a rate; a growing gap between training and validation accuracy signals overfitting, calling for more regularization or more data, while poor performance on both with a small gap signals underfitting, calling for a larger or deeper model. The Deep Learning textbook by Goodfellow, Bengio, and Courville frames why this diagnostic work is unavoidable: deep learning optimization departs from classical optimization because the true objective, generalization performance, is intractable to measure directly, so training instead optimizes a surrogate loss and relies on validation-based early stopping; the book also notes that in high-dimensional networks, saddle points vastly outnumber genuine local minima, since the expected ratio between the two grows exponentially with the number of parameters, which is one reason a stalled loss curve more often reflects a saddle point or an ill-conditioned loss surface than a true minimum. For hyperparameters like learning rate and regularization strength, CS231n's notes recommend random search over grid search, since random sampling explores the few hyperparameters that actually matter far more efficiently, typically sampling on a logarithmic scale and staging the search from short, coarse runs over a wide range toward longer, fine runs over a narrowed range.