Information Retrieval, Question Answering, and Dialogue

Modern search begins with the inverted index, a data structure that maps every distinct term in a document collection to the list of documents containing it, which is what makes it possible to find candidate matches for a query without scanning every document in the collection. Ranking those candidates by relevance is the harder problem, and the dominant classical solution is term weighting based on term frequency and inverse document frequency: a term that appears often in a specific document but rarely across the collection as a whole is a strong signal of that document's relevance to a query containing that term, while a term appearing in nearly every document, such as a common function word, carries little discriminative value. Robertson and Zaragoza formalized and extended this intuition into the probabilistic relevance framework, which models the probability that a document is relevant to a query given its term statistics, and derived from it the BM25 ranking function. BM25 refines simple term weighting with two additional considerations: term frequency saturation, meaning that after some point additional occurrences of a query term in a document add diminishing evidence of relevance rather than linearly increasing it, and document length normalization, which prevents unfairly long documents from scoring higher purely by containing more words.

BM25 remains a strong, computationally cheap baseline still used inside many production search systems today, either alone or as one signal blended with others, precisely because it requires no training data and generalizes robustly across domains. Dense or neural retrieval methods represent an alternative approach: rather than matching on exact term overlap, they encode both queries and documents into continuous vector representations trained so that semantically related queries and passages end up close together in that vector space, which lets a retrieval system find relevant documents even when they use different words than the query — synonymy that pure term-matching methods handle poorly. In practice, many modern systems combine both: a fast sparse method like BM25 narrows a huge collection to a manageable candidate set, and a more expensive neural re-ranker orders that smaller set by finer-grained relevance judgments, a two-stage design that balances retrieval speed against ranking quality.

Extractive question answering reframes the task as a span-prediction problem: given a question and a passage known to contain the answer, the model predicts the start and end position of the answer span within that passage, rather than generating an answer word by word from scratch. Rajpurkar and colleagues built the Stanford Question Answering Dataset (SQuAD) specifically to drive progress on this formulation, crowdsourcing over 100,000 question-answer pairs where each answer is a verifiable segment of text drawn from a corresponding Wikipedia passage, with human performance on the task measured at 86.8% F1 to give researchers a concrete target. Devlin and colleagues' BERT model, pretrained on large unlabeled text using a bidirectional masked-language-modeling objective that conditions on both left and right context simultaneously in every layer, was then fine-tuned for span prediction on SQuAD with only a lightweight additional output layer, and this simple adaptation of a general-purpose pretrained model achieved a 93.2 Test F1 on SQuAD v1.1, a substantial jump over prior task-specific architectures and a demonstration that broad pretraining, not task-specific engineering, was becoming the dominant lever for progress in question answering.

Extractive QA has a hard limitation: it can only answer questions when the answer literally appears as a contiguous span somewhere in a passage the system has already been handed. Open-domain question answering removes that assumption by requiring the system to first find relevant passages from an enormous document collection (retrieval) and only then extract or generate an answer (reading), a two-stage retriever-reader pipeline. Lewis and colleagues generalized this further with retrieval-augmented generation (RAG), which combines a pretrained sequence-to-sequence generator (parametric memory, encoded in the model's own weights) with a neural retriever that pulls relevant passages from an explicit document index such as Wikipedia (non-parametric memory) at inference time. Because the retrieved passages are supplied directly as context rather than baked permanently into model weights, RAG models can incorporate new or updated information without retraining, and the authors showed the approach set a new state of the art on several open-domain QA benchmarks while producing more specific and factually grounded text than a purely parametric generator working from memory alone.

Weizenbaum's ELIZA, built in 1966, is the field's founding case study in both the power and the danger of surface-level pattern matching. Running as a script called DOCTOR that simulated a Rogerian psychotherapist, ELIZA worked by decomposing a user's typed input using rules triggered by specific keywords, then reassembling a response from a template associated with the matched rule — for instance, transforming "I am unhappy" into a prompt like "why do you say you are unhappy?" There was no semantic understanding whatsoever inside the program; it never modeled meaning, only surface syntax and stored patterns. What surprised and troubled Weizenbaum was that many users attributed genuine understanding, and even empathy, to the program anyway, projecting comprehension onto a system that had none — a phenomenon now commonly called the ELIZA effect, and an early warning that a system's apparent conversational competence is not reliable evidence of any underlying understanding.

Contemporary dialogue systems split broadly into task-oriented systems, designed to help a user accomplish a specific goal such as booking a flight or resetting a password through a constrained set of intents and slots, and open-domain conversational agents, which aim to sustain more general, flexible conversation. Modern open-domain and knowledge-grounded dialogue increasingly reuses the retrieval-augmented generation pattern from question answering: rather than relying purely on facts implicitly encoded in a language model's parameters during training, a dialogue system can retrieve relevant passages from an external, updatable knowledge source and condition its generated response on them, which helps ground answers in verifiable text and reduces (though does not eliminate) the tendency to state fluent, confident falsehoods. Even so, evaluating open-domain dialogue quality remains a genuinely unresolved research problem: there is no single automatic metric that reliably captures coherence, factual grounding, helpfulness, and appropriateness simultaneously, so human evaluation, imperfect and expensive as it is, remains standard practice rather than a stopgap awaiting replacement.