Autoencoders and Variational Autoencoders

An autoencoder is a neural network trained to reconstruct its own input, structured as two parts: an encoder that maps the input x to a latent code z, typically of much lower dimensionality than x, and a decoder that maps z back to a reconstruction x-hat, with the whole network trained end to end to minimize a reconstruction loss such as mean squared error between x and x-hat. If z has strictly lower dimensionality than x — an undercomplete autoencoder — the network cannot simply learn the identity function, because the bottleneck forces information loss; to reconstruct well despite this constraint, the encoder must learn to keep only the information in x that is most useful for reconstruction and discard the rest. When the encoder and decoder are linear and the loss is squared error, this reduces to essentially the same objective solved by principal component analysis; with nonlinear encoder and decoder networks, an autoencoder can learn a nonlinear, curved manifold that captures the data's structure far more flexibly than any fixed linear subspace could.

A bottleneck in dimensionality is not the only way to prevent an autoencoder from learning a trivial identity mapping. A denoising autoencoder is trained on a different task entirely: the input is deliberately corrupted (for example by adding random noise or masking parts of it), and the network is asked to reconstruct the original, clean version from the corrupted input. Because the corruption process destroys information that a pure copying strategy would need, the network is forced to learn the underlying structure of the data well enough to fill in or correct for the corruption, which tends to produce more robust and useful representations than an undercomplete autoencoder trained on uncorrupted inputs alone — the network cannot simply pass pixels through unchanged, since the corrupted pixels it receives are, by construction, not the pixels it must output. Other regularization strategies, such as penalizing the latent code to encourage sparsity, serve a similar purpose when the latent dimensionality is not smaller than the input's, discouraging trivial solutions by a different route.

Trained autoencoders have several practical uses beyond straightforward compression. The learned latent code can serve as a lower-dimensional feature representation for downstream tasks, effectively performing unsupervised dimensionality reduction that a supervised model further downstream can then build on. Because the network has learned to reconstruct well only the kind of data it was trained on, it also reconstructs unfamiliar, out-of-distribution inputs relatively poorly, which makes the reconstruction error itself a usable signal for anomaly detection — an input the network struggles to reconstruct is, in effect, an input that does not resemble the patterns the network learned during training. What a vanilla autoencoder does not guarantee, however, is any particular structure in the latent space beyond low reconstruction error at the training points: nothing forces nearby latent codes to decode to similar-looking outputs, or forces the latent space to be densely and smoothly filled, which makes sampling a random latent point and decoding it an unreliable way to generate new, realistic examples — the gap that variational autoencoders were designed to close.

A variational autoencoder (VAE) reframes the encoder and decoder in explicitly probabilistic terms, rather than treating them as arbitrary deterministic functions fit only to minimize a reconstruction error. Instead of the encoder outputting a single deterministic latent code for a given input, it outputs the parameters of an approximate posterior distribution over latent codes, q(z|x) — in the standard formulation, the mean and (log) variance of a diagonal Gaussian — and the decoder defines a generative distribution p(x|z) over the data given a latent code. The underlying goal is maximizing the marginal likelihood p(x) that the generative model assigns to the training data, but that quantity requires integrating over all possible latent codes and is intractable to compute directly for any reasonably expressive decoder; the VAE sidesteps this by instead maximizing a tractable lower bound on it, the evidence lower bound (ELBO), which can be optimized with standard stochastic gradient methods rather than exact inference.

The ELBO decomposes into two interpretable terms. The first is a reconstruction term, the expected log-likelihood of the true input x under the decoder's distribution, given a latent code sampled from the encoder's approximate posterior — this term rewards the model for reconstructing its input well, playing a role analogous to the reconstruction loss in an ordinary autoencoder. The second is a Kullback–Leibler (KL) divergence term that penalizes how far the approximate posterior q(z|x) diverges from a fixed prior distribution over the latent space, typically a standard normal distribution. This KL term acts as a regularizer: it pulls every input's posterior distribution toward the same shared prior, which prevents the encoder from spreading different inputs' latent codes arbitrarily far apart or leaving large unused gaps in the latent space, and it is precisely this regularization, entirely absent from a plain autoencoder's reconstruction-only loss, that gives the VAE's latent space its distinctive, well-behaved geometry.

Training this model with gradient descent requires backpropagating through the sampling step z ~ q(z|x), but sampling is not, by itself, a differentiable operation with respect to the distribution's parameters — there is no way to ask how a random draw would have changed had mu shifted slightly, at least not directly. The paper's key technical contribution, the reparameterization trick, resolves this by rewriting the sampling operation as a deterministic, differentiable function of the encoder's outputs and an independent source of randomness: z = mu + sigma ⊙ epsilon, where mu and sigma are the encoder's outputs and epsilon is drawn from a fixed standard normal distribution, unrelated to any learned parameters. Because mu and sigma now appear as ordinary differentiable operations in this expression, gradients of the ELBO with respect to the encoder's parameters can be computed by standard backpropagation, exactly as they would be in any other feedforward computation, making the entire encoder–decoder system trainable end to end with stochastic gradient descent, with the randomness confined entirely to the independent epsilon term.

The KL regularization term in the VAE objective has a direct geometric consequence: because every input's approximate posterior is pulled toward the same shared prior distribution, the latent codes for different inputs end up occupying a densely packed, roughly continuous region of latent space rather than scattered clusters separated by empty, meaningless gaps. This is what makes latent-space interpolation meaningful in a trained VAE in a way it typically is not in a plain autoencoder: taking the latent codes for two real examples, walking along a path between them (for instance, a straight line), and decoding several points along that path tends to produce a sequence of outputs that changes smoothly and plausibly from one example to the other, because the intermediate latent points fall in a region the decoder has actually learned to interpret sensibly, rather than in an unused void the decoder never encountered during training — a direct payoff of the same KL term that also keeps individual inputs from being encoded arbitrarily far apart.

This same regularized structure is what enables the model to generate entirely new examples rather than only reconstructing or interpolating between existing ones. Since training pulls the aggregate distribution of encoded latent codes toward the prior — typically a standard normal distribution — sampling a fresh latent vector directly from that same prior and passing it through the decoder tends to produce a plausible, novel output, without ever encoding a real example first, which is exactly the generative capability the hospital team's interpolation use case depends on. This is precisely the capability a plain autoencoder lacks: because nothing in its training objective encourages the latent space to be densely filled or to follow any particular distribution, a latent point drawn at random usually falls in a region the decoder has never learned to interpret meaningfully, and the resulting output tends to be unrealistic or degenerate, since the decoder was never trained on, or near, that part of the space.

In practice there is a tunable tradeoff between how much weight the reconstruction term and the KL term each carry in the training objective. Weighting the KL term more heavily pushes the encoder's posteriors closer to the shared prior, which tends to produce a smoother, more regularized latent space at some cost to how precisely individual inputs can be reconstructed; weighting it less heavily favors sharper, more faithful reconstructions at the cost of a latent space that is less smoothly and evenly filled. Choosing this balance is a genuinely open design decision that depends on whether an application cares more about faithful reconstruction of specific inputs, favoring the hospital team's compression use case, or about smooth, controllable generation and interpolation across the latent space, favoring their interpolation use case — the same underlying architecture can be tuned toward either priority depending on how the two loss terms are weighted against each other, and it remains an active area of variation in VAE-based architectures beyond the basic formulation covered here.