Dimensionality Reduction and Latent-Variable Models

Principal Component Analysis has two classic and mathematically equivalent formulations, developed roughly three decades apart. Karl Pearson's 1901 paper "On Lines and Planes of Closest Fit to Systems of Points in Space" posed the problem geometrically: given a cloud of points in multidimensional space, find the line (or, more generally, plane) that minimizes the sum of squared perpendicular distances from every point to that line — a best-fit subspace rather than a best-fit prediction. Three decades later, Harold Hotelling, motivated by a psychometric problem of reducing several correlated measures of educational ability (such as reading speed and arithmetic speed) to a smaller number of underlying constructs, reformulated the same idea algebraically: find the directions of maximum variance in the data, computed as the eigenvectors of the data's covariance matrix, ordered by their corresponding eigenvalues from largest to smallest. Minimizing reconstruction error and maximizing captured variance turn out to be the same optimization problem, viewed from two different angles.

Concretely, given mean-centered data, PCA computes the covariance matrix, finds its eigenvectors (the principal component directions) and eigenvalues (the variance captured along each direction), and projects the data onto the top few eigenvectors to obtain a lower-dimensional representation. As a worked example: for the four mean-centered 2D points (2, 1), (−2, −1), (1, −1), and (−1, 1), the covariance matrix works out to [[2.5, 0.5], [0.5, 1.0]]. Its eigenvalues solve λ² − 3.5λ + 2.25 = 0, giving λ₁ ≈ 2.65 and λ₂ ≈ 0.85 (their sum, 3.5, matches the matrix's trace, and their product, 2.25, matches its determinant — a useful arithmetic check). The first principal component therefore captures λ₁ ⁄ (λ₁ + λ₂) ≈ 75.8% of the total variance, and the second the remaining 24.2%.

This variance-based view is why PCA is such a common first step before visualization, compression, or feeding data into a downstream model: the eigenvalues directly quantify how much information is lost by keeping only the top k components, which is typically displayed as a "scree plot" of eigenvalues in decreasing order, or summarized as a cumulative explained-variance curve. PCA is also mathematically equivalent to a truncated singular value decomposition of the (mean-centered) data matrix, which is how it is actually computed in practice for large datasets rather than by explicitly forming the covariance matrix. Its core limitation is that it is a purely linear projection — it can only capture structure that is well-described by straight lines and flat subspaces, a restriction that motivates both the probabilistic latent-variable models and the nonlinear manifold-learning methods covered later in this module.

PCA is a deterministic geometric projection with no explicit model of noise: it simply finds the subspace that best summarizes the observed variance, with no distinction between "signal" variance worth keeping and "noise" variance that should be discarded. Factor analysis instead poses a probabilistic generative model: each observed variable is assumed to be a linear combination of a smaller number of unobserved (latent) factors, plus its own independent noise term. This means factor analysis explicitly separates variance that is shared across observed variables (attributed to common latent factors) from variance that is unique to each individual variable, whereas PCA's components can mix both kinds of variance together. The two methods often produce numerically similar results when noise is small and roughly uniform across variables, but they answer different questions: PCA asks "what low-dimensional subspace best reconstructs the data," while factor analysis asks "what small number of hidden causes, plus independent noise, could plausibly have generated the observed correlations."

Casting dimensionality reduction as a probabilistic generative model — rather than a deterministic projection — opens the door to a much larger family of latent-variable models, a framework covered extensively in canonical treatments of pattern recognition and machine learning. In this framework, one specifies a prior distribution over latent variables and a conditional distribution describing how observed data is generated from those latents, then infers the latent variables (and any model parameters) using the observed data, typically via the EM algorithm or its variational generalizations. This generative perspective on dimensionality reduction is what allows the same underlying idea to scale far beyond simple linear factor models into modern deep generative approaches.

The Variational Autoencoder, introduced by Kingma and Welling, is the natural deep-learning extension of this idea: rather than a linear mapping from latent factors to observed variables, a VAE uses a neural network "decoder" to generate observed data from latent variables, and a second neural network "encoder" to approximate the (otherwise intractable) posterior distribution over those latents given an observation. Because sampling from a probability distribution is not differentiable, Kingma and Welling introduced the reparameterization trick, which rewrites the sampling step so that gradients can flow through it via standard backpropagation, allowing the whole model to be trained end-to-end by maximizing a tractable lower bound on the data likelihood. The result is a nonlinear, deep-learning generalization of the same latent-variable idea that motivated classical factor analysis three-quarters of a century earlier: compress high-dimensional observations into a smaller set of latent variables from which the original data can plausibly be regenerated.

PCA and factor analysis are fundamentally linear: they assume the data's true structure lies close to a flat subspace. Many real datasets instead lie on or near a curved, nonlinear manifold — imagine photographs of a single object rotating through many angles, which trace out a smooth curved surface in pixel space even though each individual photo is a very high-dimensional vector. Isomap, introduced by Tenenbaum, de Silva, and Langford, addresses this by replacing straight-line Euclidean distance with an approximation of geodesic distance — the distance measured along the manifold's surface rather than cutting through the space it is embedded in. Concretely, Isomap builds a neighborhood graph connecting each point to its nearby points (using either a fixed radius or a fixed number of nearest neighbors), approximates the geodesic distance between any two points as the length of the shortest path through this graph, and then applies classical multidimensional scaling to that matrix of geodesic distances to produce a low-dimensional embedding that preserves them as faithfully as possible.

Locally Linear Embedding, developed independently by Roweis and Saul in the same year, takes a different but related strategy that avoids the shortest-path computation entirely. LLE first reconstructs each data point as a weighted linear combination of its nearest neighbors, choosing weights that are invariant to rotation, rescaling, and translation of the local neighborhood, which means these weights capture purely local geometric relationships. It then solves a second optimization problem: find low-dimensional coordinates for every point such that the same reconstruction weights, applied to the new low-dimensional neighbors, still reconstruct each point as accurately as possible. Because both steps reduce to sparse eigenvalue problems rather than iterative search, LLE is not prone to getting stuck in local minima, and its only major free parameter is the number of neighbors used to define each local neighborhood.

t-SNE, developed by van der Maaten and Hinton, targets a somewhat different goal: producing visually interpretable two- or three-dimensional embeddings for exploration rather than preserving distances in a way suitable for downstream numerical analysis. It converts high-dimensional Euclidean distances into a probability distribution over which points are "neighbors," using a Gaussian centered on each point, and seeks a low-dimensional embedding whose corresponding neighbor probabilities — computed using a heavy-tailed Student-t distribution rather than a Gaussian — match the high-dimensional ones as closely as possible, measured by Kullback-Leibler divergence and minimized via gradient descent. The Student-t distribution's heavier tails specifically solve what the authors call the "crowding problem": in a low-dimensional space there simply isn't enough room to place many moderately distant points near a common point without them overlapping, and the heavier tails let genuinely close points stay close while pushing moderately distant points further apart. A practical caution for all three methods, but especially t-SNE, is that they preserve local neighborhood structure well but distort global distances and cluster sizes — the distance between two well-separated clusters in a t-SNE plot is not a reliable measure of how different those clusters actually are.