Recurrent and Sequence Neural Networks

Many of the most important prediction problems in artificial intelligence are not about isolated, fixed-size inputs but about sequences: a sentence is a sequence of words, a stock history is a sequence of prices, a sensor log is a sequence of readings. A feedforward network processes each input independently through a fixed set of weights and has no built-in notion of order or memory — feeding it a sentence requires either truncating or padding it to a fixed length and discarding the fact that word five relates to word two. Recurrent neural networks (RNNs) address this by introducing a hidden state vector that is updated at every time step and passed forward to the next one. At each step t, the network computes a new hidden state h_t as a function of the current input x_t and the previous hidden state h_{t-1}, typically h_t = tanh(W x_t + U h_{t-1} + b), and the same weight matrices W and U are reused at every position in the sequence. This weight sharing across time is what allows an RNN to process sequences of arbitrary length with a fixed number of parameters, and it gives the network, in principle, a mechanism for carrying information forward indefinitely through the recurrent connection.

Training a recurrent network requires an extension of standard backpropagation called backpropagation through time (BPTT). Conceptually, the recurrent network is "unrolled" into a long feedforward computation graph with one copy of the network per time step, each copy sharing the same weights, and ordinary backpropagation is then run on this unrolled graph. Because the same weight matrices appear at every time step, the total gradient with respect to those weights is a sum of contributions from every step, and the gradient flowing backward from a late time step to an early one must pass through a long chain of repeated multiplications by the same Jacobian matrix (and, along the way, repeated multiplications by the derivative of the activation function). When the relevant eigenvalues of that Jacobian are consistently below one in magnitude, the product shrinks geometrically with the number of time steps and the gradient vanishes; when they are consistently above one, the product grows geometrically and the gradient explodes. Both failure modes make it extremely difficult for a plain RNN trained with gradient descent to learn dependencies that span more than a modest number of time steps, even though the recurrent architecture is mathematically capable of representing such dependencies.

The practical consequence is that vanilla RNNs tend to do well on tasks where the relevant context is nearby — predicting the next character given the last few characters, for instance — but degrade sharply as the distance between a cue and the information that depends on it grows. Natural language is full of exactly this kind of long-range structure: subject–verb agreement can span an entire relative clause, and resolving what a pronoun refers to can require remembering a noun introduced many words earlier, as in the assistant example above. Time-series problems have analogous long memory requirements, such as a seasonal pattern that only repeats after many time steps. This mismatch between what plain recurrence can represent in theory and what it can actually learn in practice, given gradient-based training, is the problem that gated recurrent architectures were designed to solve, and it motivates the cell-state and gating mechanisms introduced in the next lesson.

The Long Short-Term Memory (LSTM) architecture was introduced specifically to overcome the vanishing gradient problem in recurrent networks. Its central idea is to give the network a separate memory pathway, the cell state c_t, that is updated mostly by addition rather than by repeated multiplication through a nonlinearity. At each time step, an LSTM cell computes three gate vectors using sigmoid activations — a forget gate f_t, an input gate i_t, and an output gate o_t — along with a candidate update g_t computed with a tanh activation. The cell state is then updated as c_t = f_t ⊙ c_{t-1} + i_t ⊙ g_t, where ⊙ denotes elementwise multiplication: the forget gate decides what fraction of the old memory to keep, and the input gate decides how much of the new candidate content to write in. The hidden state exposed to the rest of the network is then a gated, tanh-squashed view of this cell state, h_t = o_t ⊙ tanh(c_t). Because the forget-gate pathway from c_{t-1} to c_t is a near-linear, elementwise operation rather than a matrix multiplication followed by a saturating nonlinearity, gradients can flow backward along the cell state across many time steps without shrinking geometrically, so long as the forget gate stays open — an architectural property sometimes described as a constant error carousel.

The three gates are themselves small learned networks: each is a sigmoid applied to a linear combination of the current input and the previous hidden state, so the network learns, as part of training, when to remember, when to overwrite, and when to expose its memory to downstream computation. This gives the LSTM a qualitatively different failure mode than the plain RNN: instead of being architecturally forced to blend all past information together with a fixed decay rate, it can learn to hold a specific piece of information almost unchanged across many steps (forget gate near 1, input gate near 0) and then release or overwrite it precisely when needed. This is what allows an LSTM-based language model to carry the gender or number of a subject across an intervening clause, updating its cell state only when new, relevant information actually arrives rather than passively decaying every step. Because the gates are learned jointly with the rest of the network rather than hand-specified, the very same LSTM cell architecture can, depending on the task and the data it is trained on, settle into very different memory strategies — holding some features almost permanently stable while refreshing others at nearly every step — purely as a consequence of the gradients produced by whatever loss the network is trained to minimize.

The Gated Recurrent Unit (GRU) was proposed shortly afterward as part of an encoder–decoder architecture for statistical machine translation, and it simplifies the gating idea. A GRU merges the cell state and hidden state into a single vector and uses only two gates: a reset gate that controls how much of the previous hidden state contributes to computing a new candidate, and an update gate that interpolates between the previous hidden state and that candidate to produce the new hidden state. Where the LSTM maintains a dedicated memory channel, gated separately from the hidden state exposed to the rest of the network, the GRU collapses these into a single vector serving both roles, updated by one interpolation gate rather than by separate forget and input gates acting on separate quantities. With fewer parameters and no separate output gate, GRUs are cheaper to train than LSTMs while achieving comparable performance on many sequence tasks in practice, and the choice between the two architectures is often driven by empirical tuning on the task at hand — dataset size, typical sequence length, and available compute — rather than by any settled theoretical advantage of one over the other.

Many sequence tasks benefit from seeing the entire input before making any prediction, rather than only the input seen so far. A bidirectional RNN addresses this by running two separate recurrent networks over the same sequence — one processing it left to right, the other right to left — and concatenating their hidden states at each position. The resulting representation at position t incorporates context from both before and after t, which is valuable for tasks like part-of-speech tagging or named-entity recognition, where the correct label for a word can depend on words that come later in the sentence — for instance, disambiguating a word that could be either a noun or a verb often requires seeing what follows it, not just what precedes it. Bidirectionality does not solve the vanishing gradient problem by itself; it is typically combined with gated cells (bidirectional LSTMs or GRUs) to get both full-context awareness and long-range memory in the same model, and because both directions can be computed independently before being combined, bidirectional processing adds relatively little extra design complexity on top of the underlying gated cell.

A second major architectural pattern built on recurrent cells is the sequence-to-sequence (seq2seq) encoder–decoder model, introduced for tasks like machine translation where the input and output are both sequences, possibly of different lengths, with no requirement that words align one to one between them. An encoder RNN reads the entire source sequence and compresses it into a single fixed-length vector, usually its final hidden state. A separate decoder RNN is then initialized with this vector and generates the output sequence one token at a time, feeding each generated token back in as input to produce the next one, continuing until it produces a designated end-of-sequence marker. This factorization let a single differentiable model be trained end to end to map an input sequence directly to an output sequence, replacing the hand-engineered, multi-stage pipelines that dominated machine translation before it, and it established the encoder–decoder pattern that later architectures, including attention-based and Transformer models, would build directly on top of.

The fixed-length bottleneck at the heart of the basic seq2seq design is also its main limitation. Whatever the length of the source sequence — five words or fifty — the encoder must squeeze everything relevant into one vector of fixed dimensionality, and empirically translation quality degrades noticeably as source sentences get longer, because early information gets overwritten or diluted by the time the encoder reaches the end of the sequence and produces the vector the decoder will rely on for the entire output. This is precisely the problem that motivated researchers to let the decoder look back at all of the encoder's intermediate hidden states rather than only the final one, weighting each one differently at each decoding step depending on what is currently being generated. That mechanism, attention, removes the fixed-length bottleneck entirely, restores the decoder's access to every part of the source sequence at every step, and is the subject explored in depth in the next module.