Computer Vision

Computer vision research organizes the problem of understanding images into a small set of well-defined tasks, and Stanford's CS231n course frames these as a natural progression in specificity. The simplest is image classification, where a system is given a fixed set of category labels and must assign exactly one label to an entire input image, for example deciding whether a photograph shows a cat, a dog, or a car. Object detection goes further by asking the system to find every instance of relevant objects within an image and to output both a class label and a bounding box, a rectangular region specifying that object's location, meaning a single photo can produce several labeled boxes rather than one whole-image label. Segmentation pushes specificity to the pixel level: semantic segmentation assigns a category label to every pixel in the image without distinguishing between separate instances of the same class, so two adjacent cars might both simply be labeled car and blend into one region, while instance segmentation additionally separates distinct objects of the same class into individually delineated regions. This taxonomy matters practically because different applications need different granularities of output, and choosing the wrong task formulation can make an otherwise good model useless for the job at hand.

A self-driving car system illustrates why this taxonomy is not just academic. Classification alone could tell an engineer that a pedestrian is present somewhere in the frame, but it cannot say where, so the vehicle would have no way to know how to steer or brake. Object detection solves that by returning bounding boxes around each pedestrian, cyclist, and vehicle along with their class labels, giving the planning system concrete spatial coordinates to reason about. Semantic segmentation adds a further layer of understanding by classifying the drivable road surface, sidewalks, and sky pixel by pixel, which is essential for tasks like lane-boundary estimation where a rectangular box would be far too coarse. Instance segmentation becomes necessary when the system must track two overlapping pedestrians in a crowd as distinct entities rather than one merged blob, since safety-critical decisions often depend on knowing precisely how many separate agents are present and where each one's boundary lies. Real production vision stacks frequently run several of these task types in parallel or in sequence, using the coarser and cheaper classification and detection models to narrow attention before applying more expensive per-pixel segmentation to the regions that matter most.

Underlying all of these tasks, convolutional neural networks, or CNNs, extract visual features through a hierarchy of successive convolutional layers, a structure that CS231n's course notes describe directly in terms of what individual filters learn to detect. Early layers, operating on raw pixel values, tend to learn filters that activate on simple, low-level patterns such as edges of a particular orientation or blotches of a particular color, essentially rediscovering the kinds of primitive visual features that classical computer vision once hand-engineered. As the network deepens, each successive layer combines the outputs of the previous one over a larger spatial region, so filters in the middle of the network respond to more complex textures and motifs, such as combinations of edges that form corners, curves, or repeating patterns like a honeycomb or wheel-like arrangement. By the final convolutional layers, filters can respond to recognizable object parts, such as a wheel, an eye, or a wing, and these part-level responses are then combined by later fully connected or pooling layers into representations that support whole-object recognition. This edges-to-textures-to-parts-to-objects progression means the network is not memorizing pixel patterns for each training image but building an increasingly abstract, increasingly invariant representation of visual structure, which is a large part of why the same trained backbone network can often be reused, or fine-tuned, as the foundation for classification, detection, and segmentation heads alike.

Progress on the vision tasks described in the previous lesson depended heavily on the availability of large, carefully labeled datasets, and the single most influential such resource is ImageNet, introduced by Deng and colleagues in their 2009 paper 'ImageNet: A Large-Scale Hierarchical Image Database.' ImageNet was built by organizing images according to the WordNet hierarchy of nouns and collecting large numbers of hand-verified example images for each concept, aiming eventually to cover tens of millions of images spanning tens of thousands of categories. This scale and label quality was a deliberate departure from earlier, much smaller benchmark datasets, and it was motivated by the observation that many machine learning algorithms, especially the higher-capacity models researchers wanted to explore, were starved for sufficiently large and diverse training data. Because ImageNet organized concepts hierarchically, from broad categories down to fine-grained distinctions, it also made it possible to design benchmarks at varying levels of difficulty, from separating dogs from cars down to distinguishing specific dog breeds from one another. The dataset's creation is widely regarded as a foundational infrastructure contribution to the field, on par with the algorithmic advances it later helped enable, because it gave researchers a common, large-scale testbed against which competing approaches could be fairly compared.

Starting in 2010, a subset of ImageNet was used to run the ImageNet Large Scale Visual Recognition Challenge, or ILSVRC, an annual competition that became the de facto benchmark for measuring progress in image classification and related tasks. The classification track of ILSVRC used roughly 1.2 million training images spanning 1,000 object categories, with a held-out evaluation set of images that competing systems had never seen, and it scored entries using a top-5 error rate, meaning a prediction counted as correct if the true label appeared anywhere among the model's five most confident guesses. Because every team trained and evaluated on the same data split under the same rules, ILSVRC results year over year gave the community an unusually clean, apples-to-apples signal of whether new techniques were actually working better than previous ones, rather than relying on scattered claims across incompatible datasets. In the years leading up to 2012, most competitive ILSVRC entries relied on hand-engineered visual features, such as SIFT or Fisher Vector encodings, combined with classifiers like support vector machines, and progress on the leaderboard was correspondingly incremental.

The 2012 ILSVRC classification competition marked a turning point for the entire field. The winning entry, a deep convolutional neural network submitted by Krizhevsky, Sutskever, and Hinton and commonly referred to as AlexNet, achieved a top-5 error rate of 15.3 percent on the competition's held-out test set, according to the official ILSVRC 2012 results published on the ImageNet Large Scale Visual Recognition Challenge site. The second-place entry that year, built primarily from combinations of hand-engineered SIFT, LBP, GIST, and color-SIFT features feeding classical classifiers, achieved a top-5 error rate of 26.2 percent, meaning AlexNet's error rate was less than 60 percent of its nearest competitor's, an unusually large margin of victory for an established benchmark that previous years' entries had been improving only gradually. This single result is widely cited as the moment deep convolutional networks demonstrated, under rigorous and public competitive conditions, a dramatic and verifiable improvement over the best non-deep-learning computer vision techniques of the time. The result reshaped research priorities across the field: within a few years, nearly every top ILSVRC entry and much of mainstream computer vision research had shifted to deep CNN architectures built on the same core ingredients AlexNet combined, namely large labeled datasets like ImageNet, convolutional architectures, and GPU-accelerated training.