Semi-Supervised, Self-Supervised, and Weakly Supervised Learning

Fully supervised learning requires every training example to carry a human-verified label, but obtaining such labels is often the most expensive part of building a machine learning system — while raw, unlabeled data is comparatively cheap and abundant. Semi-supervised learning tries to exploit both: a small pool of labeled examples together with a much larger pool of unlabeled ones. This is only possible because of structural assumptions about how the data is organized — for instance, the "cluster assumption," that points in the same natural cluster tend to share the same label, or the closely related "manifold assumption," that data lies on a lower-dimensional manifold along which labels vary smoothly. Under these assumptions, the geometry revealed by the unlabeled data — where the dense regions and gaps in feature space actually are — can sharpen the decision boundary a classifier draws, even though the unlabeled points themselves carry no label information directly.

One of the simplest semi-supervised strategies is self-training (also called pseudo-labeling): train a classifier on the small labeled set, use it to predict labels for the unlabeled examples it is most confident about, add those newly "pseudo-labeled" examples to the training set, and repeat. A more structured alternative, formalized by Blum and Mitchell as co-training, applies when each example can be described by two distinct feature views that are each individually sufficient to predict the label but are not perfectly correlated with one another — for example, the text of a web page and the anchor text of links pointing to it. Co-training trains one classifier per view on the small labeled set, then has each view's classifier label the unlabeled examples it is most confident about and add those to the other view's training set, exploiting the fact that a mistake made confidently by one view is unlikely to be repeated by an independent, differently-informed view.

Both self-training and co-training are heuristic in the sense that they can reinforce a classifier's own early mistakes if the model's initial confidence is miscalibrated, so semi-supervised learning also includes more theoretically grounded approaches — generative models that jointly model the distribution of features and labels, transductive methods that only try to label the specific unlabeled points at hand rather than build a general-purpose classifier, and graph-based label propagation that spreads labels across a similarity graph connecting labeled and unlabeled points. What unites the family is the core insight that unlabeled data is not informationless — it reveals the shape of the underlying data distribution, and that shape constrains which labelings of the data are plausible even before a single label is observed.

Self-supervised learning goes further than semi-supervised learning by discarding the need for any human-provided labels at all: instead, it constructs a "pretext task" whose supervisory signal is automatically derived from the raw, unlabeled data's own structure. One of the clearest and most influential early examples is word2vec, introduced by Mikolov, Chen, Corrado, and Dean, which learns dense vector representations of words purely from large volumes of unlabeled text by training a model to predict a word from its surrounding context (or the reverse) — no human ever labels a single word, yet the resulting word vectors capture rich syntactic and semantic regularities, and the approach scaled efficiently enough to train on 1.6 billion words in under a day. The insight generalizes far beyond language: any transformation, corruption, or partial view of data that a model can be trained to predict, reconstruct, or recognize creates a "free" supervisory signal, with the ultimate goal of learning representations useful for downstream tasks the model was never explicitly trained on.

In computer vision, one of the most effective self-supervised strategies is contrastive learning, exemplified by SimCLR from Chen, Kornblith, Norouzi, and Hinton. SimCLR generates two different augmented views of the same image (through random cropping, color distortion, and blurring) and trains a network so that the two augmented views of the same image ("positive pairs") are pulled close together in representation space, while views of different images in the same training batch ("negative pairs") are pushed apart. The authors found that the specific composition of augmentations used, an additional learnable nonlinear projection layer between the representation and the contrastive loss, and training with larger batches and longer schedules were all critical to representation quality, ultimately reaching 76.5% top-1 accuracy on ImageNet using only a linear classifier trained on top of frozen self-supervised features — matching a fully supervised ResNet-50 — and substantially outperforming a classic supervised baseline like AlexNet even when fine-tuned on just 1% of the labels.

The through-line connecting word2vec and SimCLR, despite operating in completely different modalities, is that both convert an unsupervised prediction or discrimination problem — predict the context, or recognize which augmented views came from the same source image — into a supervisory signal rich enough to shape genuinely useful representations. Because the choice of pretext task, augmentation strategy, and training objective all substantially affect what a self-supervised model actually learns to represent, and because these choices interact differently across modalities such as text, images, audio, and structured data, self-supervised learning remains a highly active area of research rather than a settled, one-size-fits-all technique.

A third strategy sidesteps the labeling bottleneck differently: rather than using no labels (self-supervised) or a small set of clean labels plus unlabeled data (semi-supervised), weak supervision embraces labels that are cheap to obtain but individually noisy, indirect, or incomplete, and combines many such noisy sources to approximate the quality of hand-labeled data. Sources of weak supervision include heuristic rules written by domain experts (for example, "if a clinical note contains the word 'malignant' near a tumor mention, label the case positive"), distant supervision from external knowledge bases, crowdworker annotations of uncertain quality, or even the outputs of older, less accurate models. Any individual weak source may be wrong on a substantial fraction of examples, but if multiple weak sources make different kinds of errors, their combined signal can still be far more reliable than any single source alone.

Snorkel, developed by Ratner and colleagues, formalizes this into a system built around the "data programming" paradigm: rather than labeling individual examples by hand, a domain expert writes a set of labeling functions — small heuristic programs that vote on a label (or abstain) for each example, with unknown and potentially correlated accuracies. Snorkel then fits a generative model over the labeling functions' outputs to estimate each function's accuracy and how correlated different functions are with one another, without ever needing ground-truth labels to do so, and uses these estimates to combine the noisy votes into a single probabilistic training label for each example. In controlled studies, subject matter experts using Snorkel built comparably accurate models roughly three times faster than seven hours of manual labeling would have taken, with average performance gains of 45.5%, and in real deployments — including with government health agencies — the resulting models came within a few percentage points of models trained on large hand-curated datasets.

Weak supervision trades label quality for label quantity and speed, and this tradeoff carries genuine risks: labeling functions written by domain experts can encode the same systematic blind spots or biases those experts hold, and errors correlated across multiple weak sources (for instance, several heuristics all relying on the same superficial keyword) will not be automatically detected by a generative model that assumes sources are only partially correlated. Because of this, weakly supervised pipelines require careful auditing of labeling functions and their failure modes, and the theoretical question of how best to model correlations among an arbitrary, growing set of weak sources remains an active area of machine learning research rather than a fully solved problem.