Natural Language Processing

Long before a computer can extract meaning from a sentence, it must first break that sentence down into discrete pieces that a program can manipulate, and this initial decomposition is the job of tokenization. A tokenizer scans a raw character stream and decides where one word, punctuation mark, or symbol ends and the next begins, converting an undifferentiated string such as Marie was born in Paris into a sequence of tokens: Marie, was, born, in, Paris, and a period. Toolkits such as Stanford CoreNLP formalize this as the first stage of a broader annotation pipeline, one that also performs sentence splitting to determine where one sentence ends and the next begins within a longer document. CoreNLP describes its overall purpose as deriving linguistic annotations for text, including token and sentence boundaries, parts of speech, named entities, and dependency and constituency parses, which makes explicit that tokenization is the foundation every later stage depends on. Because downstream components such as taggers and parsers operate on tokens rather than raw characters, errors introduced at the tokenization stage, such as splitting a contraction or a hyphenated compound incorrectly, tend to propagate and degrade the accuracy of everything that follows. This sequential, stage-by-stage design, where each annotator consumes the output of the ones before it and adds a new layer of linguistic structure, is the defining architecture of what is often called the classic NLP pipeline.

Once text has been tokenized, the next classic pipeline stage is part-of-speech tagging, which assigns a grammatical label to every token in the sentence, marking it as a noun, verb, adjective, preposition, and so on. Stanford CoreNLP's documentation describes this annotator as assigning part-of-speech labels to tokens, such as whether they are verbs or nouns, and it notes that every token in a sentence receives a tag, so no word is left unlabeled. In CoreNLP's own worked example, the sentence Marie was born in Paris is tagged so that the token Marie receives the label NNP, the standard Penn Treebank tag for a singular proper noun. This tagging step is inherently disambiguating: a word like book is a noun in I read the book but a verb in please book the flight, and the tagger must use surrounding context to choose the correct label for each occurrence. Because many downstream tasks, including parsing and named entity recognition, rely on knowing whether a token is functioning as a noun, verb, or modifier, accurate part-of-speech tagging is treated as a prerequisite rather than an optional add-on in a classic NLP pipeline. The output of this stage is therefore not just a bag of words but a sequence of tokens each annotated with its grammatical role, a substantially richer representation than the raw tokenized text alone.

Building on tokenized, part-of-speech-tagged text, named entity recognition identifies spans of tokens that refer to specific real-world entities and classifies them into categories. Stanford CoreNLP's documentation explains that its NER annotator recognizes named entities such as person and company names in text, and that its default English configuration distinguishes twelve entity classes grouped into named entities such as PERSON, LOCATION, ORGANIZATION, and MISC, numerical entities such as MONEY, NUMBER, ORDINAL, and PERCENT, and temporal entities such as DATE, TIME, DURATION, and SET. Under the hood, CoreNLP combines several machine learning sequence models, described in its documentation as a combination of conditional random field sequence taggers trained on labeled corpora, together with rule-based components for interpreting numbers and dates. The final classic pipeline stage, parsing, goes further still by analyzing how the tagged tokens relate to one another grammatically, producing either a constituency parse that groups words into nested phrases or a dependency parse that draws directed grammatical links, such as subject or object, between individual words. CoreNLP's overview page lists dependency and constituency parses among the linguistic annotations the toolkit can derive, situating parsing as the stage that turns a flat, tagged sequence of tokens into an explicit tree or graph structure representing the sentence's grammar. Taken together, tokenization, part-of-speech tagging, named entity recognition, and parsing form a layered pipeline in which each stage adds structure that the next stage, and eventually an application built on top of it, can exploit.

For decades, one of the central problems in NLP was how to represent the meaning of a word in a form a computer program could use, since a word is, to a machine, initially nothing more than an arbitrary string of characters. Stanford's CS224N course notes describe the field's answer to this problem as distributional semantics, the idea of representing the meaning of a word based on the contexts in which it usually appears, a principle commonly summarized by the observation that words occurring in similar contexts tend to have similar meanings. Under this distributional hypothesis, two words such as dog and puppy are considered similar in meaning not because a dictionary says so but because they tend to be surrounded by similar neighboring words across large amounts of text. Early attempts to operationalize this idea produced sparse, high-dimensional representations, such as one-hot vectors or raw co-occurrence counts between words, which explicitly recorded which words appeared near which other words but scaled poorly and did not generalize well to unseen contexts. The shift from these sparse counting-based methods toward dense, learned representations is precisely what set the stage for word embedding models, which compress the distributional information in a word's contexts into a comparatively short vector of real numbers. That shift proved foundational, because a dense vector, unlike a sparse count table, can be fed directly into downstream statistical and neural models as a compact numerical feature.

The most influential early implementation of this idea was Word2Vec, introduced by Mikolov, Chen, Corrado, and Dean in their 2013 paper Efficient Estimation of Word Representations in Vector Space. Rather than counting co-occurrences directly, Word2Vec trains a shallow neural network to predict words from their context, and in doing so it learns, as a byproduct, dense vector representations for every word in the vocabulary. The paper and the accompanying CS224N course notes describe Word2Vec as comprising two related architectures, continuous bag-of-words, or CBOW, which predicts a center word from its surrounding context words, and skip-gram, which does the reverse by predicting the surrounding context words from a given center word. Mikolov et al. reported that these architectures could learn high-quality word vectors from a data set of 1.6 billion words in under a day, a substantial efficiency gain over the neural language models that had come before, and that the resulting vectors achieved strong results on word similarity and analogy benchmarks. Once trained, these embeddings placed semantically related words near one another in the vector space, so that words used in similar contexts, such as synonyms or words from the same topical domain, ended up with similar vector representations. Word2Vec's combination of computational efficiency and representational quality made dense word embeddings a standard input representation for a wide range of NLP models throughout the mid-2010s.

Word embeddings like Word2Vec had an important limitation, however: each word received exactly one fixed vector regardless of how it was used, so bank had the same representation in river bank and bank account. Stanford's CS224N course materials trace the field's response to this limitation, describing how deep learning approaches went on to obtain very high performance across many NLP tasks using single end-to-end neural models that did not require the traditional, task-specific feature engineering that classic pipelines relied on. This trajectory culminated in Transformer-based architectures, which generate contextualized representations that change depending on a word's surrounding sentence, and, as the CS224N course site puts it, in the 2020s amazing further progress was made through the scaling of large language models, such as ChatGPT. As a result, the current CS224N curriculum moves from foundational word vectors and recurrent networks in its early weeks to Transformers, pretraining, and cutting-edge large language model techniques in its later weeks, reflecting how the center of gravity in NLP research has moved. One practical consequence of this shift is that many applications no longer run tokenization, part-of-speech tagging, named entity recognition, and parsing as separate, hand-assembled pipeline stages the way classic toolkits like CoreNLP do; instead, a single pretrained Transformer-based model is often fine-tuned or prompted to perform a target task directly. Tokenization has not disappeared, since even large language models must first split text into subword units before processing it, but explicit POS tagging, NER, and parsing as discrete, separately trained steps are now comparatively less central to mainstream NLP systems than they were in the era before contextualized, Transformer-based representations became dominant.