Object Detection, Segmentation, and Tracking

Object detection requires solving two problems at once: classifying what objects are present, and localizing where each one is with a bounding box. Two-stage detectors, epitomized by Ren and colleagues' Faster R-CNN, split this into a proposal stage and a classification stage but made the proposal stage itself learned and fast: their Region Proposal Network (RPN) is a fully convolutional network that shares image features with the downstream detection network and simultaneously predicts, at every position across the image, both an "objectness" score (is there likely to be an object here at all?) and coordinate adjustments for candidate boxes of various scales. Because the RPN shares convolutional features with the classifier that scores the final proposals, region proposals become "nearly cost-free," and the whole system, evaluated on PASCAL VOC and MS COCO, achieved state-of-the-art accuracy for its time while reaching about 5 frames per second on a GPU using only 300 proposals per image — a considerable improvement in speed over prior region-proposal methods that relied on slow, separate, non-learned proposal algorithms.

Redmon and colleagues' YOLO (You Only Look Once) took a more radical departure: rather than proposing regions and then classifying them, it reframes detection as a single regression problem, where one neural network predicts all bounding boxes and their class probabilities directly from the full image in a single forward pass. This unified design made YOLO dramatically faster — the base model ran at 45 frames per second, and a lighter "Fast YOLO" variant reached 155 frames per second, fast enough for genuinely real-time applications where earlier two-stage detectors struggled. This speed came with a tradeoff the authors were explicit about: YOLO tended to make more localization errors, meaning its predicted boxes were sometimes less precisely placed, but produced markedly fewer false positives (falsely detecting an object where none exists) than competing systems. Both families of detector rely on the same core evaluation concept to decide whether a predicted box counts as correct: Intersection over Union (IoU), the ratio of the overlapping area between a predicted and a ground-truth box to the total area the two boxes cover together, and non-maximum suppression, a post-processing step that discards redundant overlapping boxes predicted for the same object, keeping only the highest-confidence one.

Detection draws a rectangular box around an object; segmentation instead assigns a label to every individual pixel, which is a fundamentally finer-grained task. Semantic segmentation labels each pixel with a class (road, person, sky) without distinguishing between separate instances of the same class, while instance segmentation goes further, separately delineating each individual object instance even when multiple objects of the same class overlap or sit adjacent to one another. Ronneberger, Fischer, and Brox's U-Net, originally designed for biomedical image segmentation where labeled training data is scarce and expensive to produce, established an architecture that has since become foundational well beyond medicine: a contracting encoder path that progressively captures broader spatial context by downsampling, paired with a symmetric expanding decoder path that upsamples back to the original resolution for precise per-pixel localization, connected by skip connections that pass high-resolution detail directly from encoder to decoder layers at matching scales, compensating for detail otherwise lost during downsampling. U-Net demonstrated it could train effectively even with very few annotated images, and using this architecture won the ISBI cell tracking challenge in its transmitted light microscopy categories while running fast enough to segment a 512-by-512 image in under a second on a contemporary GPU.

Instance segmentation requires going beyond a per-pixel class label to also separate distinct object instances, and He, Gkioxari, Dollár, and Girshick's Mask R-CNN accomplished this by extending Faster R-CNN with a third parallel output branch: alongside the existing branches for bounding box regression and object classification, a new branch predicts a binary segmentation mask for each detected object region, running in parallel with — rather than as a downstream step after — the existing detection machinery. This addition came at only modest extra computational cost, letting the full system run at roughly 5 frames per second, and Mask R-CNN achieved top performance across all three tracks of the COCO benchmark suite simultaneously — instance segmentation, bounding-box detection, and human keypoint (pose) detection — using a single unified framework, outperforming the specialized, single-model 2016 COCO challenge winners across every one of those tracks at once.

Tracking asks a question detection and segmentation alone cannot answer: given detections in every frame of a video, which detection in frame two corresponds to the same physical object as which detection in frame one? The dominant paradigm, tracking-by-detection, runs an off-the-shelf per-frame object detector (such as Faster R-CNN or YOLO) on every video frame independently, then solves a separate data-association problem to link detections across frames into consistent object tracks. Bewley and colleagues' SORT (Simple Online and Realtime Tracking) demonstrated that this association problem could be solved effectively using only classical, non-learned techniques: a Kalman filter predicts each existing track's expected position in the next frame based on its estimated velocity, and the Hungarian algorithm then finds the optimal one-to-one matching between predicted track positions and the new frame's actual detections, typically using IoU between predicted and detected boxes as the matching cost. Despite this simplicity, SORT matched the accuracy of contemporary state-of-the-art trackers while running at 260 Hz — over 20 times faster — and the authors showed detection quality itself was the single largest lever on overall tracking performance, since tracking can only ever be as good as the detections it links.

SORT's purely motion-based association breaks down under a common and difficult scenario: occlusion, where one object passes behind another and briefly disappears from detection, after which the two objects' predicted positions may become ambiguous or their identities may get swapped once both reappear — an identity switch. Wojke, Bewley, and Paulus's DeepSORT addressed this directly by adding an appearance-based association signal alongside the existing motion model: a deep metric-learning network, trained offline on person re-identification data, produces an appearance embedding for each detected object, and during tracking these embeddings are compared via nearest-neighbor matching to help re-associate an object with its correct track even after a period of occlusion, which the authors reported reduced identity switches by 45% while preserving the real-time speed suitable for high-frame-rate applications. Robust tracking through long occlusions, crowded scenes, and rapidly changing appearance nonetheless remains a genuinely active area of research rather than a fully solved problem, particularly in dense crowds or when objects of very similar appearance move close together.