Attention Mechanisms and Transformers

The fixed-length bottleneck of the basic seq2seq encoder–decoder model — the requirement that an entire source sequence be compressed into one vector before decoding begins — was addressed by an attention mechanism proposed for neural machine translation. Instead of discarding the encoder's intermediate hidden states after computing a single summary vector, the attention-based model keeps all of them, one per source position, and lets the decoder consult all of them anew at every decoding step. Concretely, at each step the decoder computes an alignment score between its current state and each encoder hidden state, using a small learned scoring function; these scores are passed through a softmax to produce a set of normalized attention weights that sum to one; and a context vector for that step is formed as the weighted sum of the encoder hidden states, using those weights. This context vector, different at every decoding step, is then combined with the decoder's own state to produce the next output.

This design directly removes the fixed-length bottleneck: rather than forcing all source information through one vector regardless of sentence length, the model can, in principle, spread its attention weights over as many source positions as it needs, and it can put weight on different source positions for different output words, revisiting earlier positions as many times as necessary across the course of generating the full output sequence. Because the attention weights form a genuine probability distribution over source positions, they are also interpretable in a way that a single fixed vector is not — visualizing the weights the model assigns when producing a given output word shows, quite directly, which source words it is drawing on at that moment, and in trained translation models these weights tend to align closely with the intuitive word-by-word correspondences a bilingual reader would draw between the two sentences, even though the model was never given explicit alignment labels during training.

It is useful to describe this mechanism in the more general vocabulary that later architectures adopted: a query (the decoder's current state), a set of keys (the encoder hidden states, used to compute alignment scores), and a set of values (the encoder hidden states again, this time used to compute the weighted sum). Framed this way, attention is a differentiable lookup: given a query, retrieve a weighted combination of stored values, where the weights come from how well the query matches each corresponding key, rather than retrieving a single stored value by an exact, discrete match the way a dictionary lookup would. This query–key–value framing generalizes cleanly beyond the original encoder–decoder setting where it first appeared — nothing in the mechanism actually requires the queries to come from a decoder or the keys and values to come from a separate encoder — and it is exactly the abstraction that the Transformer architecture builds its entire design around, as the next lesson develops in detail.

Self-attention applies the query–key–value mechanism within a single sequence rather than between an encoder and a separate decoder: every position in a sequence computes a query, a key, and a value from its own representation, and then attends over the keys and values produced by every other position in that same sequence, including itself. The Transformer architecture formalizes this as scaled dot-product attention: given matrices Q, K, and V (queries, keys, and values, one row per position), the output is computed as softmax(QK^T / sqrt(d_k)) V, where d_k is the dimensionality of the key vectors. The dot product QK^T measures how well each query matches each key; dividing by sqrt(d_k) counteracts the tendency of dot products to grow large in magnitude as dimensionality increases, which would otherwise push the softmax into regions with extremely small gradients; and the softmax turns the scaled scores into a normalized weighting over values, exactly as in the earlier attention mechanism, but now computed between every pair of positions in the same sequence rather than between a decoder state and an encoder sequence.

Rather than computing a single attention operation, the Transformer computes several in parallel — multi-head attention. The queries, keys, and values are each linearly projected into several lower-dimensional subspaces (heads), scaled dot-product attention is applied independently within each subspace, and the resulting outputs are concatenated and linearly projected back to the model's working dimensionality. Because each head has its own learned projections, different heads can specialize in attending to different kinds of relationships within the same sequence — for instance, one head might learn to track short-range syntactic dependencies while another tracks longer-range coreference-like relationships — and the model combines these different views rather than being restricted to a single fixed notion of relevance. Splitting one large attention computation into several smaller, independently projected ones also keeps the total computational cost comparable to a single full-dimensional attention operation, since each head operates on a proportionally smaller subspace, so multiple heads are essentially free in exchange for this added representational flexibility.

A pure self-attention layer, however, is permutation-invariant: swapping the order of two positions in the input and swapping the corresponding rows of the output leaves the attention computation itself unchanged, so the mechanism has no inherent sense of sequence order the way a recurrent network does by construction, where each step's computation is necessarily built from the step before it. The Transformer restores this information explicitly through positional encoding: a fixed vector, computed from sinusoidal functions of the position index and the dimension index, is added to each position's input embedding before the first attention layer, so position information enters the model as part of the input representation itself rather than through the architecture's connectivity. Because these functions are deterministic and defined for any position, this scheme lets the model represent relative position through predictable, learnable relationships between the sinusoids at different offsets, and lets it extrapolate, at least in principle, to sequence lengths not seen during training.

The full Transformer architecture stacks multiple identical layers into an encoder and a decoder. Each encoder layer consists of a multi-head self-attention sublayer followed by a position-wise feedforward network (applied identically to every position), with residual connections around each sublayer and layer normalization applied after each — design choices that make it practical to stack many such layers without training becoming unstable. Each decoder layer has an additional third sublayer: after its own self-attention (which must be masked so that a given output position cannot attend to positions after it, preserving the autoregressive property needed to generate output one token at a time) the decoder performs cross-attention, in which its queries come from the decoder itself but its keys and values come from the encoder's output. This cross-attention sublayer is a direct descendant of the original attention mechanism between decoder and encoder states, now expressed in the same query–key–value form as everything else in the architecture.

The central practical advantage of self-attention over recurrence is parallelism. In an RNN, computing the hidden state at position t requires the hidden state at position t-1 to already be available, which forces strictly sequential computation across the length of the sequence during both training and inference. In a self-attention layer, the representation at every position can be computed simultaneously, because each position's output depends only on the (already available) queries, keys, and values of the whole sequence, not on the output of neighboring positions computed within the same layer. This lets Transformer layers be computed as large matrix multiplications that map efficiently onto parallel hardware, which is a major reason Transformer-based models could be trained on much larger datasets in much less wall-clock time than comparably sized recurrent models. The tradeoff is computational: because every position attends to every other position, a single self-attention layer costs computation and memory that scale quadratically with sequence length, which becomes a real constraint for very long sequences.

Beyond machine translation, the task the architecture was originally introduced for, the encoder and decoder stacks generalize as building blocks that can be used separately as well as together. An encoder stack alone, trained to build rich contextual representations of an input sequence, is well suited to tasks that consume a whole sequence and produce a single judgment or per-position labeling, since every position can attend freely to every other position with no ordering restriction. A decoder stack alone, with its masked self-attention already restricting each position to only see earlier positions, is naturally suited to autoregressive generation, producing one output token at a time conditioned only on what has been produced so far, without requiring a separate encoded sequence to attend to at all. This flexibility — encoder-only, decoder-only, or full encoder–decoder configurations, all built from the same underlying self-attention block with only the masking pattern and the presence or absence of cross-attention changing between them — is a significant part of why the Transformer became a general-purpose architecture for sequence modeling well beyond its original translation setting.