Retrieval-Augmented Generation and Generative AI Applications

Any retrieval system starts from the same basic problem: given a query and a large corpus of candidate passages, decide which passages are actually relevant. Older lexical retrieval methods score this relevance by matching literal terms, which struggles when a relevant passage uses different words for the same idea, a synonym, a paraphrase, or a related concept that never shares an exact keyword with the query. Dense retrieval takes a different approach: it represents both the query and every candidate passage as a dense vector, or embedding, positioned in a shared numerical space such that texts with similar meaning end up near one another, regardless of whether they share any exact words. Relevance is then scored with a similarity measure over these vectors, most commonly cosine similarity, which asks how closely two vectors point in the same direction rather than how large they are. This reframes retrieval as a geometry problem — finding the vectors nearest a query vector — which is what makes it tractable to search over enormous, constantly growing text collections.

Karpukhin et al.'s 2020 paper on Dense Passage Retrieval, or DPR, is an influential concrete implementation of this idea for open-domain question answering. DPR uses a dual-encoder architecture: one BERT-based encoder converts questions into embeddings, and a separate BERT-based encoder converts passages into embeddings, and the two are trained together with a contrastive objective so that a question's embedding lands close to embeddings of passages that actually answer it and far from embeddings of unrelated passages used as negative examples. Karpukhin et al. reported that this dense approach substantially outperformed traditional sparse lexical retrieval on open-domain QA retrieval benchmarks. To see the geometry concretely: suppose a query embedding is q = (1, 0), and three candidate passage embeddings are d1 = (0.8, 0.6), d2 = (0, 1), and d3 = (-1, 0). Cosine similarity for d1 is (1×0.8 + 0×0.6) / (1×1) = 0.8; for d2 it is (1×0 + 0×1) / (1×1) = 0; for d3 it is (1×-1 + 0×0) / (1×1) = -1. Ranking by similarity, d1 is retrieved first, d2 second, and d3 — pointing in the opposite direction from the query — last.

Real corpora used for retrieval hold millions or billions of passage embeddings, and computing exact similarity between a query and every single one of them at search time is too slow for interactive use. FAISS, described by Johnson, Douze, and Jégou, addresses this with approximate nearest-neighbor search: techniques such as product quantization compress each embedding into a much smaller representation, and index structures narrow the search to a promising subset of candidates rather than scanning the entire corpus, trading a small, controllable amount of retrieval accuracy for large gains in search speed and memory footprint, with GPU-accelerated implementations further increasing throughput. This approximate-search layer, sitting between a dense retriever like DPR and a downstream generator, is what makes retrieval over large, real-world corpora practically fast enough to sit inside an interactive system rather than a slow offline batch job — at the cost of an index that must itself be built, stored, and kept up to date as the underlying corpus changes.

Retrieval on its own only returns a ranked list of passages; retrieval-augmented generation, introduced by Lewis et al. in 2020, connects that retrieval step to a generator so the two work together as one system. A RAG system combines a pretrained sequence-to-sequence generator with a non-parametric retrieval component: given a query, it first retrieves the top-k most relevant passages from an indexed corpus, then conditions the generator on both the original query and the retrieved passages, so the generated text can draw directly on that retrieved evidence rather than relying solely on whatever the generator's parameters happened to memorize during pretraining. This has two practical advantages over a purely parametric model generating from memory alone: the underlying knowledge source can be updated or swapped by changing the retrieval index, without retraining the generator itself, and because the retrieved passages are visible, outputs can in principle be checked or traced back against the specific evidence the generator was given, an important, if imperfect, mitigation for hallucination in knowledge-intensive tasks.

Retrieving text is one way to extend a generator beyond its frozen parameters; calling external tools is another. Schick et al.'s 2023 Toolformer trains a model to use tools, such as a calculator, a search engine, or a calendar, in a self-supervised way: the model itself proposes candidate API calls to insert at various points in training text, actually executes those calls, and keeps only the insertions that measurably improve its ability to predict the text that follows, discarding the rest — meaning it learns when and how to invoke a tool without needing hand-labeled tool-use demonstrations. Yao et al.'s ReAct takes a complementary approach for multi-step tasks: rather than a single forward pass, the model interleaves explicit natural-language reasoning steps with actions, such as issuing a search query, and observations, the results that action returns, so it can revise its plan mid-task based on what a tool actually reports back, rather than committing to one fixed reasoning chain up front. Together, retrieval and tool calling let a generator's output be grounded not just in retrieved text but in the results of actions it takes during generation itself.

Two further application patterns extend generation in different directions. Structured generation constrains a model's output to match a fixed schema, such as a specific set of JSON fields an application expects, which matters whenever generated text needs to be consumed programmatically rather than read by a person — a practical engineering layer on top of the free-text generation capability described above, rather than a change to the underlying model. Multimodal generation follows a related logic in a different modality: Rombach et al.'s 2021 latent diffusion models compress images into a much lower-dimensional latent space using a learned autoencoder, then run the iterative, computationally expensive diffusion denoising process in that compact latent space rather than directly on raw pixels, substantially reducing the compute required to train and sample from high-resolution text-to-image models. Architectures, safety mitigations, and evaluation methods for multimodal generation are all still moving quickly, and — like generative-model evaluation more broadly — this remains an actively evolving area of research rather than a settled one.