Generative AI

Machine learning models can broadly be split into two families based on what they learn from data. A discriminative model learns a decision boundary or a conditional probability P(y given x): given an input x, such as an image, it predicts a label y, such as 'cat' or 'dog'. It never tries to understand how the input itself was produced; it only cares about separating classes as cleanly as possible. A generative model instead tries to learn something about the joint distribution or the data distribution P(x) itself, capturing the underlying statistical structure of the training examples closely enough that the model can produce new samples that look like they belong to that same distribution. This is a fundamentally harder task: instead of drawing one boundary line between categories, the model must internalize the shape, texture, and structure of an entire class of data well enough to synthesize plausible new instances of it. Classifying a photo of a handwritten digit as a '7' is discriminative work; producing a brand-new, convincing image of a handwritten '7' that never existed in the training set is generative work.

In 2014, Ian Goodfellow and colleagues proposed Generative Adversarial Networks, or GANs, as a new framework for training generative models through an adversarial game between two neural networks. The first network, the generator, takes a vector of random noise as input and transforms it into a synthetic sample, such as an image, attempting to make that sample indistinguishable from real training data. The second network, the discriminator, is trained as a binary classifier that looks at a sample and tries to determine whether it came from the real training set or was produced by the generator. Goodfellow et al. frame training as a minimax game with a value function that the discriminator tries to maximize, by getting better at telling real from fake, while the generator simultaneously tries to minimize it, by getting better at fooling the discriminator. The two networks are trained together, with the discriminator's gradients providing the learning signal that pushes the generator toward producing more realistic samples over time.

The paper's key theoretical result is that, under idealized conditions where both networks have enough capacity and are trained to convergence, this adversarial process has a unique optimal solution: the generator's distribution over samples matches the true data distribution exactly, and at that point the discriminator can do no better than randomly guessing, correctly identifying real versus fake samples only half the time. Goodfellow et al. describe this equilibrium using an analogy to counterfeiters and police: the generator is like a counterfeiter trying to produce fake currency that passes inspection, while the discriminator is like a detective trying to catch the fakes, and the competition between the two drives both to improve until the counterfeits become indistinguishable from genuine currency. In practice, training this minimax game is notoriously difficult to stabilize, since the generator and discriminator must improve at a roughly matched pace, but the framework proved enormously influential because it required no explicit likelihood function for the data and instead let the discriminator implicitly supply the training signal through the adversarial competition.

Variational Autoencoders, introduced by Diederik Kingma and Max Welling in their 2013 paper 'Auto-Encoding Variational Bayes', take a different, more probabilistically grounded approach to generative modeling. A VAE consists of two neural networks: an encoder, which maps an input, such as an image, to the parameters of a probability distribution, typically a Gaussian, over a lower-dimensional latent space, and a decoder, which takes a point sampled from that latent distribution and reconstructs it back into something resembling the original input. Because directly computing the true posterior distribution over latent variables is generally intractable, Kingma and Welling introduce a recognition model, the encoder, that approximates this posterior, and they train the whole system by optimizing a quantity called the variational lower bound, or evidence lower bound, on the data's log-likelihood, which balances how well the decoder reconstructs the input against how closely the encoder's approximate distribution matches a simple prior, usually a standard normal distribution, over the latent space.

A central technical contribution of the paper is the reparameterization trick, which allows the random sampling step inside the encoder to be rewritten so that gradients can still flow through it during backpropagation, making the whole encoder-decoder system trainable end to end with standard stochastic gradient methods. Once trained, new data can be generated by sampling a random point from the prior distribution over the latent space and passing it through the decoder alone, without needing the encoder at all. Because the latent space is explicitly regularized to resemble a smooth, continuous distribution, VAEs tend to produce a well-organized latent space where nearby points decode to similar-looking outputs, which is useful for interpolating between samples, though VAE-generated images have historically tended to look somewhat blurrier than samples from GANs, a tradeoff often attributed to the reconstruction term in the objective favoring averaged, safer predictions.

Diffusion models, formalized in their modern form by Jonathan Ho, Ajay Jain, and Pieter Abbeel in their 2020 paper 'Denoising Diffusion Probabilistic Models', or DDPM, take yet another approach, built around two Markov chain processes. The forward process gradually destroys structure in a real data sample by repeatedly adding small amounts of Gaussian noise over many timesteps, until the sample becomes indistinguishable from pure random noise. The reverse process is a neural network trained to undo this corruption one small step at a time, learning to predict and remove the noise added at each step so that, starting from pure noise, it can iteratively reconstruct a realistic sample; Ho, Jain, and Abbeel show that training this network reduces to a simple objective where it repeatedly learns to predict the noise component that was added at a given timestep. Because each denoising step only has to make a small, tractable correction rather than generating an entire realistic sample in one shot, diffusion models produce highly detailed, high-fidelity outputs, which is a major reason they became the dominant approach behind later high-resolution image generation systems; this same forward-noising and learned-reverse-denoising principle, refined and scaled by later work, underlies the general diffusion approach used in systems such as DALL-E 2 and Stable Diffusion, though the specific architectural details of those later systems extend well beyond what the original DDPM paper itself describes.