Representation, Embedding, and Contrastive Learning

A local or one-hot representation assigns each distinct item — each word in a vocabulary, for instance — its own dedicated dimension, so that any two distinct items are, by construction, maximally dissimilar (their vectors have zero overlap) regardless of how semantically related they actually are. A distributed representation instead encodes each item as a pattern of activity across many shared dimensions, where each dimension can participate in representing many different items and each item is described by many dimensions at once. This matters because it allows the representation to generalize by combinatorial sharing of statistical strength: if a model has learned that a particular dimension tends to be active for positive-sentiment words, it can extend that pattern to a word it has seen only a handful of times, as long as that word's other dimensions place it near other positive-sentiment words it has seen more often. A central theme in representation learning is that a great deal of a system's downstream performance is determined less by the learning algorithm applied on top of a representation than by the quality of the representation itself — how well it disentangles the underlying factors that actually explain variation in the data.

Word embeddings are a concrete and influential instance of learned distributed representations. Rather than hand-designing features for each word, an embedding model learns a dense vector for every word in a vocabulary purely from the statistics of how words co-occur in large amounts of raw text, with no manual labeling required. Two closely related architectures were proposed for this: continuous bag-of-words (CBOW), which predicts a target word from the words surrounding it in a context window, and skip-gram, which does the reverse, predicting the surrounding context words from a single target word. Both architectures deliberately avoid any deep, expensive computation per training example, which is what let them be trained efficiently on very large text corpora. Despite being trained on a shallow, simple prediction objective, the resulting embedding vectors end up organized so that words used in similar contexts land near each other in the vector space, and the geometry of the space captures certain semantic and syntactic regularities that were never explicitly specified as training targets.

The idea of learning a dense vector representation from a self-defined predictive objective, rather than from human-provided labels, generalizes well beyond individual words. The same basic pattern — train an encoder to map a raw input (a word, an image, a sentence, a graph node) to a fixed-size vector using some auxiliary task computable from unlabeled data — reappears across representation learning broadly, and the resulting embeddings can typically be reused across many different downstream tasks rather than needing to be relearned from scratch for each one, which is a major reason embeddings became a standard first stage in many machine learning pipelines. A downstream classifier trained on top of frozen, pre-learned embeddings often needs far less labeled data than one trained on raw input features, because the embedding has already absorbed a great deal of structure from the much larger pool of unlabeled data it was trained on, leaving the classifier with a comparatively easy job.

Metric learning takes a more direct approach to shaping an embedding space: rather than relying on a representation's geometry to end up useful as a side effect of some other prediction task, it defines a training objective explicitly in terms of distances between embeddings. The classic formulation uses triplets — an anchor example, a positive example that should be considered similar to the anchor, and a negative example that should be considered dissimilar — and a margin-based loss that penalizes the network whenever the negative is not sufficiently farther from the anchor than the positive is. Writing d(a, p) and d(a, n) for the anchor-to-positive and anchor-to-negative distances and m for a margin hyperparameter, a typical triplet loss is max(0, d(a, p) − d(a, n) + m): the loss is zero once the negative is already farther away than the positive by at least the margin, and otherwise it produces a gradient that pushes the positive closer to the anchor and the negative farther away, exactly proportionally to how badly the margin condition is currently violated.

Contrastive predictive coding (CPC) generalizes this intuition into a probabilistic objective usable for unsupervised representation learning, and it removes the need for the hand-picked, task-specific positive and negative pairs that a triplet-loss setup typically relies on. CPC trains an encoder together with an autoregressive model to predict a future latent representation from past context, and frames this prediction as a classification problem: given the true future representation mixed in among a batch of other, unrelated ("negative") representations drawn from elsewhere in the data, the model must identify which one is the real future. This is formalized as the InfoNCE loss, a categorical cross-entropy over the true positive versus the sampled negatives, and it can be motivated as encouraging the learned representation to retain information that is useful for predicting the future while discarding information that is not — conceptually connecting the representation-learning objective to how much predictive information the representation preserves, rather than to how well the representation supports exact reconstruction of the raw input.

A notable property of the contrastive predictive coding approach is that its underlying mechanism does not depend on any assumption specific to one data modality. The same encoder-plus-autoregressive-prediction-plus-contrastive-classification recipe was applied successfully to speech, images, natural-language text, and reinforcement learning observations in simulated three-dimensional environments, using architectures and negative-sampling details adapted to each modality — different choices of encoder network, different definitions of what counts as "future," different ways of sampling negatives — while keeping the same core contrastive objective throughout every domain. This generality is a central reason contrastive learning is treated as a general-purpose strategy for unsupervised representation learning rather than a technique specific to any one kind of data, since the mechanism only requires some notion of a related positive pair and a supply of unrelated negatives to contrast it against, with no requirement that the data be images, text, or any other particular format, and it sets up the large-scale, augmentation-based contrastive framework for images covered in the next lesson.

SimCLR is a framework for learning image representations through contrastive learning without using any labels during pretraining. Its pipeline starts by applying a composition of random data augmentations — such as cropping, resizing, and color distortion — to a single image twice, independently, producing two different "views" of the same underlying image that differ in exact pixel values but should, intuitively, still depict the same content. Both views are passed through a shared-weight base encoder network (such as a standard convolutional network) to produce representation vectors, and those vectors are then passed through a small nonlinear projection head before the contrastive loss is applied. The training objective treats the two augmented views of the same original image as a positive pair and treats every other image's augmented views within the same training batch as negatives, pulling the positive pair's projected representations together while pushing them away from the negatives, with no manual labels involved anywhere in the process.

Two design choices turned out to matter substantially for how well this simple recipe works. First, the specific composition of augmentations used to generate the two views is critical: some combinations (notably combining random cropping with color distortion) force the model to learn representations invariant to changes that are not informative about image content, which produces a much harder and more useful predictive task than weaker augmentation choices that leave the two views too easy to match trivially. Second, introducing a learnable nonlinear projection head between the base encoder's representation and the contrastive loss substantially improves the quality of the representation that ends up being useful downstream; after pretraining, this projection head is typically discarded and only the base encoder's output is kept and reused for downstream tasks, since the projection head's job was specifically to make the contrastive loss easier to optimize, not to produce the final representation that later tasks will consume.

Evaluated with a linear classifier trained on top of a frozen, pretrained SimCLR encoder — the standard "linear evaluation" protocol for judging the quality of a self-supervised representation, in which the encoder's weights are not updated at all and only the small linear classifier on top is trained — the framework reached 76.5% top-1 accuracy on ImageNet, a substantial improvement over prior self-supervised methods evaluated the same way, achieved with no class labels used anywhere during representation learning. This kind of linear-evaluation result is the standard way researchers judge whether a self-supervised representation has captured meaningful structure, since a simple linear classifier can only succeed if the underlying representation has already done most of the work of separating the classes. The framework's authors also found that contrastive learning of this kind benefits more from larger batch sizes and longer training than typical supervised learning does, since larger batches supply more negative examples per training step, making the contrastive discrimination task both harder and more informative during training, and giving the encoder a richer set of contrasts to learn from at every gradient update.