Generative AI Adaptation, Alignment, and Evaluation

A model pretrained only to predict the next token is a general-purpose text continuer, not an assistant: given an instruction, it may continue the instruction rather than obey it, since nothing in next-token prediction over raw web text specifically rewards being helpful. Instruction tuning is one response to this gap. Wei et al., in their 2021 paper introducing FLAN, phrased a large collection of existing NLP datasets — for tasks like summarization, translation, and question answering — as natural-language instructions paired with the desired response, and fine-tuned a large pretrained language model on this mixture. The key experimental question was whether this kind of tuning would generalize beyond the specific tasks it was trained on, so Wei et al. held out entire task categories during evaluation. They found that instruction tuning substantially improved zero-shot performance on these unseen task types compared to the untuned base model, with FLAN outperforming zero-shot GPT-3 on most of the datasets they tested and, on some datasets, even exceeding few-shot GPT-3's performance despite using no in-context examples at inference time. This established instruction tuning as a comparatively cheap, broadly effective first step in adapting a pretrained model toward following instructions, well before any human-preference data enters the picture.

Instruction tuning on demonstrations teaches a model to imitate example responses, but demonstrations alone cannot capture graded judgments like which of two decent responses is better. Christiano et al.'s 2017 paper on deep reinforcement learning from human preferences addressed this more general problem outside of language: rather than hand-engineering a reward function for a reinforcement learning agent, which is often difficult or impossible to specify precisely for complex behaviors, they had non-expert humans compare short pairs of agent trajectory segments and say which one they preferred, then trained a reward model to predict those comparisons and optimized the agent's policy against that learned reward. They showed this approach could train complex behaviors, including Atari game playing and simulated robot locomotion, from a comparatively small number of human comparisons, establishing preference learning as a scalable substitute for reward engineering. Ouyang et al.'s InstructGPT applied essentially this same three-stage idea to language models — supervised fine-tuning, then a reward model trained on human rankings of candidate text, then policy optimization against that reward model — turning preference learning from a general reinforcement-learning technique into the standard alignment pipeline for assistants.

The reinforcement learning stage of that pipeline is also its most operationally expensive part: it requires a separate reward model, an unstable and compute-heavy RL loop, and careful tuning to avoid the policy exploiting quirks of the learned reward. Rafailov et al.'s 2023 Direct Preference Optimization (DPO) paper showed this RL step can be skipped entirely. They proved that, for the reward objective RLHF optimizes, the optimal policy has a closed form in terms of the reference model and an implicit reward, which lets a model be trained directly on preference pairs with a simple classification-style loss — no reward model, no RL loop. Concretely: suppose β = 0.1, and for a preferred response the policy's log-probability rises from -2.5 (reference) to -2.0 (policy), a gain of 0.5 nats, while for a dispreferred response it falls from -3.0 to -3.5, a loss of 0.5 nats. Scaling each gap by β gives implicit rewards of 0.05 and -0.05, a margin of 0.10; passing that margin through the logistic function gives σ(0.10) ≈ 0.525, and the loss for this one pair is -ln(0.525) ≈ 0.644 nats — a value that shrinks toward zero as the policy learns to favor the preferred response by an ever-larger margin relative to the reference model.

Once a model can follow instructions and reflect human preferences, a further question is whether what it says is true. Lin, Hilton, and Evans's 2021 TruthfulQA benchmark targets this specifically: it consists of 817 questions across 38 categories, adversarially written so that some humans would answer falsely because of a common misconception, myth, or false belief, testing whether a model imitates human falsehoods rather than genuinely reasoning about the world. The paper reported that the best-performing model they evaluated answered truthfully only 58% of the time, compared to 94% for human performance on the same questions, and — a more unsettling finding — that the largest models in their study were generally the *least* truthful, an inverse scaling relationship: scale improves a model's ability to predict plausible-sounding continuations of text, including plausible-sounding falsehoods, without necessarily improving its grip on what is actually true. This decoupling of scale from truthfulness is a central reason factuality is treated as a distinct concern from instruction-following or preference alignment, requiring its own targeted evaluation and training effort.

Human-labeled preference data is expensive to collect at the scale needed for RLHF, and Bai et al.'s 2022 Constitutional AI paper explored reducing that dependence, specifically for harmlessness. The method has two phases. In the first, the model is prompted to critique and revise its own responses against a small set of explicit written principles — a "constitution" — producing improved responses that are then used for supervised fine-tuning without requiring a human to write or label each one. In the second phase, the model generates its own preference labels by comparing pairs of responses according to the same constitutional principles, and a preference model is trained on these AI-generated comparisons rather than purely human ones, a process the paper terms reinforcement learning from AI feedback (RLAIF). Because the guiding principles are written down explicitly rather than existing only implicitly in human labelers' judgments, Bai et al. argued this also makes the values a model is being trained toward more transparent and inspectable, alongside the practical benefit of needing far less human harmlessness labeling.

Even with instruction-tuned, preference-aligned, more-truthful models in hand, measuring which one is actually best remains genuinely unsettled. Chiang et al.'s 2024 Chatbot Arena is one influential approach: real users submit prompts to two anonymized models at once, vote for the response they prefer, and these pairwise votes are aggregated across a large volume of live, organic traffic into a relative ranking. This differs sharply from a fixed benchmark like TruthfulQA, which tests one specific capability with a static set of questions; Arena-style evaluation instead captures something closer to holistic, in-the-wild preference, at the cost of being harder to reproduce exactly and vulnerable to biases such as raters preferring longer or more confidently worded answers regardless of correctness. Evaluation methodology for generative models remains an actively evolving, unsettled area of research — there is still no single agreed-upon ground truth for what makes one model's output "better" than another's, and different evaluation methods can disagree with each other for defensible reasons.