Large Language Models

Before 2017, the dominant architectures for processing language were recurrent neural networks, including LSTMs and gated recurrent units, which read a sequence one token at a time and carried information forward through a hidden state. This sequential design created two persistent problems. First, because each step depended on the output of the previous step, training could not be parallelized across the length of a sequence, which made it slow to train on the large corpora that modern models require. Second, information about early tokens had to survive many sequential updates to influence predictions about later tokens, so recurrent models struggled to capture long-range dependencies in long documents. In their 2017 paper "Attention Is All You Need," Vaswani et al. introduced the Transformer, an architecture that dispenses with recurrence and convolutions entirely and is based solely on attention mechanisms. This design choice directly addressed both weaknesses of recurrent models by allowing every position in a sequence to be processed simultaneously during training and by giving every token a direct computational path to every other token.

The core mechanism that makes this possible is self-attention. For every token in an input sequence, the model computes three vectors known as a query, a key, and a value, each derived from the token's embedding through learned linear projections. The query of one token is compared against the keys of all tokens in the sequence, producing a set of compatibility scores that indicate how relevant each other token is to the one being processed. These scores are normalized into weights, and the token's new representation becomes a weighted sum of all the value vectors in the sequence, with more relevant tokens contributing more strongly. This lets the model directly weigh the relationship between any two tokens regardless of how far apart they are in the sequence, whether that means linking a pronoun to the noun it refers to several sentences earlier or connecting a verb to its subject across an intervening clause. Vaswani et al. additionally proposed multi-head attention, in which several attention computations run in parallel with different learned projections, allowing the model to attend to different kinds of relationships, such as syntactic structure and semantic association, at the same time.

The original Transformer described in the paper uses an encoder-decoder structure built for machine translation. The encoder is a stack of layers that each combine a multi-head self-attention sublayer with a position-wise feed-forward sublayer, processing the entire source sentence at once to build contextual representations of every input token. The decoder is a similar stack that additionally attends over the encoder's output and uses masked self-attention so that, when generating each output token, it can only look at previously generated tokens rather than tokens yet to come. Because the architecture itself has no inherent notion of token order, the model adds positional encodings to the input embeddings so that sequence position information is preserved despite processing all tokens in parallel. This combination of parallel computation, direct token-to-token attention, and explicit positional information proved so effective and so efficient to train at scale that the Transformer became the foundation not just for translation systems but for essentially all large language models that followed, including GPT-style decoder-only models that keep the masked self-attention decoder stack and drop the separate encoder.

Modern large language models are built through a two-phase process usually described as pretrain-then-adapt. In the pretraining phase, a Transformer-based model is trained on enormous quantities of text using a self-supervised objective, typically predicting the next token given all preceding tokens, so that no manually labeled data is required. Brown et al. demonstrated in their 2020 paper "Language Models are Few-Shot Learners" that scaling this simple objective up to a very large model trained on a very large corpus produces striking new capabilities. Their model, GPT-3, is an autoregressive language model with 175 billion parameters, ten times more than any previous non-sparse language model at the time. The central finding of the paper is that scaling up language models greatly improves task-agnostic, few-shot performance: GPT-3 could be shown just a handful of examples of a task written directly in its input prompt, with no gradient updates or fine-tuning at all, and would often perform competitively with models that had been explicitly fine-tuned on that task. This in-context learning capability, where the model infers what task to perform purely from the prompt, was the paper's key contribution to how practitioners think about interacting with large language models.

Before any of this text reaches the Transformer, it must be converted into a sequence of discrete tokens the model can process as numbers. The standard approach, used by OpenAI's models and documented in the tiktoken library, is byte-pair encoding, commonly abbreviated BPE. BPE builds a vocabulary by starting from individual bytes or characters and iteratively merging the most frequently co-occurring pairs into new vocabulary entries, so that common subword fragments such as "ing" or frequent whole words become single tokens while rarer words are broken into smaller pieces. This scheme is reversible and lossless, meaning tokens can always be converted back into the exact original text, and it generalizes to any input, including words the tokenizer has never seen, by falling back to smaller fragments. In practice, each token corresponds to roughly four bytes of English text on average, which is why the effective context length of a model, the number of tokens it can attend over at once, does not map directly onto a fixed number of words. Because self-attention's computational cost grows with the square of the sequence length, the choice of tokenization scheme has a direct practical effect on how much text a Transformer can process efficiently.

A model that has only been pretrained to predict the next token is not automatically a helpful assistant; it is simply a very capable predictor of what text is likely to follow a given prompt, which can include continuing an instruction rather than obeying it. Ouyang et al. addressed this gap in their 2022 paper "Training language models to follow instructions with human feedback," which introduced InstructGPT and the reinforcement learning from human feedback procedure, or RLHF, now standard for turning pretrained models into assistants. The process has three stages. First, human labelers write demonstrations of desired responses to prompts, and the pretrained model is fine-tuned on these demonstrations using ordinary supervised learning to produce a supervised fine-tuned model. Second, that model generates multiple candidate responses to a set of prompts, human labelers rank these responses by quality, and a separate reward model is trained on this dataset of comparisons to predict which of two responses a human would prefer. Third, the supervised model is further fine-tuned using reinforcement learning, with the reward model's scores as the reward signal that the policy is optimized against. Ouyang et al. reported that outputs from their 1.3 billion parameter InstructGPT model were preferred by human evaluators over outputs from the original 175 billion parameter GPT-3, despite having more than one hundred times fewer parameters, and that the procedure also improved truthfulness and reduced toxic output, demonstrating that alignment to human preferences can matter as much as raw scale.