Language Modelling, Translation, and Summarization

A language model assigns a probability to a sequence of words, and that single idea underlies everything from autocomplete to machine translation to summarization. The classical approach, the n-gram model, applies the chain rule of probability to decompose the probability of a sentence into a product of conditional probabilities, then approximates each conditional by looking back only n-1 words rather than the entire history — a bigram model conditions on one previous word, a trigram on two. These counts are estimated directly from a training corpus, but raw counting runs immediately into data sparsity: most plausible word sequences never appear in any finite corpus, so naive maximum-likelihood estimates assign zero probability to sentences a fluent speaker would find unremarkable. Smoothing techniques, which redistribute probability mass from observed to unobserved sequences, were developed specifically to address this problem, and the tradeoffs between them shaped decades of statistical natural language processing before neural methods became dominant. This history matters because it establishes the core evaluation criterion for any language model, neural or statistical: perplexity, a measure of how well a model predicts held-out text, where lower perplexity indicates the model is less "surprised" by real language.

Neural language models replace sparse count tables with dense, learned vector representations of words and a differentiable function that predicts the next word from context. Recurrent architectures, particularly the long short-term memory (LSTM) network, extended this idea to variable-length sequences by maintaining a hidden state updated at every time step, in principle allowing information from arbitrarily far back in a sentence to influence the current prediction. Sutskever, Vinyals, and Le demonstrated that this machinery could be pushed further: two multilayered LSTMs, one encoding an entire input sequence into a single fixed-dimensional vector and a second decoding a target sequence from that vector, could learn to translate English to French end to end, without any hand-built alignment or phrase tables, and were competitive with the phrase-based statistical systems that had dominated for years. They also found that a seemingly cosmetic change — reversing the order of the source sentence's words during training — measurably improved performance by shortening the average distance the model had to carry dependencies. This encoder-decoder framework, and the fixed-length "bottleneck" vector at its center, is the direct predecessor to the attention-based architectures covered next.

The encoder-decoder architecture's weakness is structural: compressing an entire source sentence, however long, into one fixed-length vector forces the model to discard information, and empirically this caused translation quality to degrade sharply as sentences grew longer. Bahdanau, Cho, and Bengio addressed this directly by allowing the decoder to "soft-search" the source sentence at each decoding step rather than relying solely on a single summary vector. Concretely, at every step the model computes a set of alignment weights over all source positions, forming a weighted sum — a context vector — that changes depending on which target word is currently being generated. Because these weights are learned jointly with translation rather than supplied by a separate alignment model, the resulting soft alignments could be visualized and, the authors showed, largely agreed with human intuitions about which source words correspond to which target words. This mechanism, attention, closed most of the performance gap with phrase-based statistical translation on English-to-French tasks and produced particularly large gains on longer sentences, precisely where the fixed-vector bottleneck had hurt most.

Attention's next transformation came from removing recurrence altogether. Vaswani and colleagues introduced the Transformer, an architecture built exclusively from attention mechanisms, with no recurrent or convolutional layers at all. Self-attention lets every position in a sequence attend directly to every other position in a single operation, which has two major consequences: dependencies between distant words no longer have to be propagated step by step through a chain of hidden states, and because there is no sequential recurrence, computation across all positions can be parallelized during training rather than processed one token at a time. Multiple attention "heads" running in parallel let the model attend to different kinds of relationships simultaneously, and positional encodings are added to the input embeddings to give the otherwise order-agnostic architecture a sense of sequence. On the WMT 2014 English-to-German benchmark the Transformer improved the best previously reported result, including ensembles, by more than two BLEU points, while training substantially faster than comparable recurrent models, and the same architecture later became the foundation for nearly all large-scale language models.

Both architectures are typically evaluated on translation with BLEU, an automatic metric based on n-gram overlap between a system's output and one or more human reference translations. BLEU correlates reasonably well with human judgment in aggregate but is a blunt instrument for any single sentence, penalizing valid paraphrases that happen not to match the reference's exact wording and rewarding outputs that game n-gram overlap without being genuinely fluent. Low-resource language pairs, domain shift, and idiomatic or culturally specific expressions remain genuinely difficult even for state-of-the-art systems, and building translation quality metrics that track human judgment more closely, rather than surface n-gram overlap, is an active area of ongoing research rather than a solved problem.

Automatic summarization splits into two families. Extractive summarization selects and stitches together existing sentences or phrases from the source document, which guarantees factual grounding in the source text but limits fluency and can produce disjointed output. Abstractive summarization instead generates new sentences that may never appear verbatim in the source, using the same sequence-to-sequence machinery covered earlier, but this flexibility introduces two persistent failure modes: factual inaccuracy, where the model states something plausible but unsupported by the source, and repetition, where the model gets stuck reproducing the same phrase. See, Liu, and Manning's pointer-generator network addressed both directly within a single architecture. A "pointer" mechanism lets the model copy words straight from the source document when appropriate, which is especially valuable for rare words, names, and numbers that a purely generative vocabulary might mishandle, while a coverage mechanism tracks which parts of the source have already been attended to and penalizes attending to them again, substantially reducing repetition. On CNN/Daily Mail news summarization, this hybrid architecture exceeded prior abstractive systems by at least two ROUGE points, a metric analogous to BLEU but built around recall of overlapping n-grams with reference summaries.

The same encoder-decoder and self-attention machinery that performs translation and summarization also underlies open-ended text generation, where a language model is used to produce novel text rather than transform an existing input. How the model converts its output probability distribution into an actual sequence of words — the decoding strategy — matters as much as the model's training. Greedy decoding always picks the single highest-probability next word and is fast but often produces bland, repetitive text; beam search tracks several high-probability partial sequences in parallel and tends to produce more fluent but sometimes overly generic output; sampling-based methods introduce controlled randomness to increase diversity at some cost to reliability. Across all of these, hallucination — a model confidently generating fluent, well-formed text that is factually wrong or unsupported by any source — remains a genuinely unresolved problem rather than a settled engineering detail, and automatic metrics like ROUGE and BLEU are known to correlate only imperfectly with human judgments of summary quality and factual accuracy, which is why human evaluation remains standard practice in serious summarization research even though it does not scale.