3D Vision, Scene Understanding, and Video Intelligence

Simultaneous Localization and Mapping (SLAM) is the foundational problem of mobile robotics and 3D vision alike: as Durrant-Whyte and Bailey describe it, SLAM is the process by which a robot builds a map of an unknown environment while simultaneously using that same map to determine its own location within it, without external positioning references like GPS. The problem is circular by construction — accurate localization normally requires a known map, and accurate mapping normally requires a known location — which is resolved probabilistically rather than by breaking the circularity outright: classical solutions represent the robot's pose and the positions of observed landmarks jointly as a probability distribution, updated incrementally as new sensor observations arrive, using techniques such as the Extended Kalman Filter (EKF-SLAM), which linearizes the nonlinear motion and observation models around a current estimate, or Rao-Blackwellised particle filters (FastSLAM), which represent the robot's trajectory with a set of weighted particles while estimating each landmark's position analytically conditioned on that trajectory. Structure-from-motion, a closely related classical technique, recovers 3D scene geometry and camera positions from a sequence of 2D images by tracking how visual features shift across viewpoints and triangulating their 3D locations from those shifts.

Neural radiance fields (NeRF), introduced by Mildenhall and colleagues, represent a markedly different strategy for the related but distinct problem of novel view synthesis: rather than reconstructing an explicit 3D structure such as a mesh or point cloud, NeRF trains a single, fully-connected neural network to represent an entire scene implicitly as a continuous volumetric function, taking a 5D input (a 3D spatial coordinate plus a 2D viewing direction) and outputting a volume density and a view-dependent color at that point. New viewpoints are then synthesized by sampling many points along camera rays passing through the scene and using classical volume rendering equations, which are differentiable, to accumulate those sampled densities and colors into a final rendered pixel — meaning the entire network can be optimized directly from a set of ordinary 2D photographs with known camera poses, with no explicit 3D geometry ever required as input. The authors demonstrated this produces photorealistic renderings that surpassed prior neural rendering techniques on complex scenes. Both SLAM and NeRF-style neural scene representations remain genuinely active research areas — SLAM in extending robustly to dynamic, changing environments, and neural radiance-field methods in scaling to real-time and large, unbounded scenes — rather than fully closed engineering problems.

Estimating human pose from an image — locating the 2D or 3D positions of a person's joints such as elbows, knees, and shoulders — becomes considerably harder when multiple people are present in a scene and must each be assigned their own correct set of joints, especially when they overlap or occlude one another. Cao and colleagues' OpenPose addressed multi-person 2D pose estimation with a bottom-up approach: rather than first detecting each individual person and then estimating that one person's pose (a top-down strategy that gets slower as more people appear), the network jointly predicts, across the whole image at once, both individual body part locations and Part Affinity Fields — a nonparametric representation encoding the direction and association between pairs of body parts that belong to the same individual — and then uses those learned associations to group detected parts into complete, correctly assigned skeletons for each distinct person. Because the core computation runs once over the entire image regardless of how many people it contains, OpenPose's processing time stayed largely constant as the number of people in a scene increased, unlike top-down methods whose runtime scales with the number of detected people, and the approach won first place in the 2016 COCO keypoints competition.

Pose estimation is one input into the broader, harder problem of holistic scene understanding: jointly reasoning about a scene's geometry (depth and layout, connecting back to the structure-from-motion and neural scene representation techniques covered earlier), its semantic content (what objects and surfaces are present, connecting to the segmentation techniques covered in the previous module), and the people and their poses within it, all at once rather than as separate, independently solved subproblems. This joint reasoning matters in practice: a robot or autonomous vehicle needs a consistent, unified understanding of where free space is, what objects and people occupy the rest of the scene, and how everything is likely to move next, not a collection of disconnected outputs from independently run detection, segmentation, and pose models that might disagree with each other. Building systems that reason jointly and consistently across geometry, semantics, and dynamics — rather than bolting together separately trained components after the fact — remains a genuinely open and actively researched problem in computer vision, without a single settled, universally adopted architecture the way object detection or semantic segmentation now have.

Video adds a temporal dimension that single-image computer vision techniques do not directly address: recognizing an action such as "opening a door" or "kicking a ball" requires interpreting how a scene changes across many consecutive frames, not just what any single frame contains. Two families of approach emerged for handling this temporal signal: two-stream architectures process a video's raw RGB frames and a separately computed optical flow field (capturing pixel-level motion between frames) through two parallel convolutional networks and combine their predictions, while 3D convolutional networks instead extend convolution itself to operate over space and time jointly. Carreira and Zisserman's Two-Stream Inflated 3D ConvNet (I3D) combined both ideas: it "inflates" filters and pooling kernels from a network already pretrained on 2D still images into 3D by adding a time dimension, letting the model start from ImageNet-pretrained weights, repurposed for video, rather than training a 3D architecture entirely from scratch on comparatively much smaller video datasets. To give this and future video models a benchmark of adequate scale and diversity, the same authors introduced the Kinetics Human Action Video dataset, substantially larger than prior standard benchmarks like UCF-101 and HMDB-51, and after pretraining on Kinetics, I3D reached 80.9% accuracy on HMDB-51 and 98.0% on UCF-101, a marked improvement attributable specifically to large-scale video pretraining.

Action recognition on short, trimmed video clips, as tackled by I3D and evaluated on Kinetics, is a meaningfully different and more tractable problem than long-form video understanding, which requires reasoning about events, causal relationships, and narrative structure unfolding over minutes rather than seconds, and doing so efficiently despite the enormous computational cost of processing video's much higher data volume compared to still images. Extending the self-attention and transformer architectures that transformed language modeling and, later, single-image vision to video adds substantial computational challenges, since naively attending over every pair of pixels across every pair of frames in a long clip scales extremely poorly, and efficient, effective architectures for long-form temporal reasoning in video remain an active area of ongoing research rather than a solved problem, honestly distinct from the comparatively more mature and standardized state of short-clip action classification.