Probability Theory for AI

Probability theory begins with a sample space — the set of all possible outcomes of an uncertain process — and assigns each event a number between 0 and 1 describing how likely it is, subject to the rule that the probabilities of all possible outcomes sum to 1 (MIT OCW 6.041, Probabilistic Systems Analysis and Applied Probability). A random variable is a function that maps outcomes of that uncertain process to numbers, letting us do arithmetic and reasoning about randomness itself rather than treating each uncertain outcome as an unstructured, one-off event. Flip a fair coin three times and let X count the number of heads observed: X can take the values 0, 1, 2, or 3, with respective probabilities 1/8, 3/8, 3/8, and 1/8 — this list of outcomes and their probabilities is exactly what is meant by the probability distribution of X, and it fully summarizes everything there is to know about how the random variable behaves.

Distributions come in two broad flavors. Discrete distributions, like the coin-flip example above, assign probability directly to individual outcomes and are common whenever an AI system counts things — the number of words in a message, the number of clicks on a recommendation, or the number of objects detected in an image. Continuous distributions, by contrast, describe variables that can take any value in a range, such as a sensor reading or a model's confidence score, and assign probability to intervals rather than single points via a probability density function; MIT's 18.05 (Introduction to Probability and Statistics) develops the Gaussian, or normal, distribution as the canonical continuous example, whose familiar bell shape places about 68% of its probability within one standard deviation of its mean and about 95% within two standard deviations, a rule of thumb used constantly when summarizing measurement uncertainty.

AI systems lean on this machinery constantly. A classifier's softmax layer outputs a categorical probability distribution over class labels, assigning each possible label a probability that must be non-negative and sum to 1, exactly mirroring the axioms above (Russell & Norvig's AIMA frames probabilistic reasoning as central to acting under uncertainty). Generative models such as diffusion models and large language models are, at their core, learned probability distributions over images or text that the system samples from to produce new outputs, one token or pixel value drawn according to the learned distribution at a time — meaning the random-variable and distribution concepts introduced here are not abstract mathematics on the side, but the literal object every probabilistic AI model is built to represent and manipulate, and every improvement in a generative model is, mathematically, an improvement in how closely its learned distribution matches the true distribution of real images or text.

Conditional probability captures how a belief about one event should update once another event is known, defined as P(A|B) = P(A and B) / P(B) (MIT OCW 6.041). Suppose an email inbox has 100 messages, of which 20 are spam and 80 are not; among the 20 spam messages, 15 contain the word 'free', while among the 80 legitimate messages, only 4 do. Of the 19 messages in total that contain 'free' (15 + 4), 15 are spam, so the conditional probability that a message is spam given that it contains 'free' is P(spam | 'free') = 15/19 ≈ 0.789 — nearly 79%, a dramatic update from the 20% baseline rate of spam in the inbox as a whole. This is exactly the kind of word-level evidence a real spam classifier combines, one word at a time, across an entire message to make its final routing decision.

Expectation, written E[X], is the probability-weighted average value a random variable takes — the long-run average outcome if the random process were repeated many times (MIT 18.05). Consider a language model's confidence that a generated sentence contains 0, 1, or 2 typos, with probabilities 0.5, 0.3, and 0.2 respectively: E[X] = (0)(0.5) + (1)(0.3) + (2)(0.2) = 0.7 typos on average, a single summary number even though no individual sentence can literally contain 0.7 typos, useful precisely because it lets many sentences be compared or ranked by one consistent number. Variance measures how spread out a random variable's outcomes are around that expectation, computed as Var(X) = E[X²] − (E[X])²; for the same distribution, E[X²] = (0)(0.5) + (1)(0.3) + (4)(0.2) = 1.1, giving Var(X) = 1.1 − 0.49 = 0.61 — a number that quantifies how much individual outcomes typically deviate from the average of 0.7 typos per sentence, distinct from the average itself, which only reports the central tendency of the distribution.

These two summary statistics are everywhere in AI practice, not just in probability theory itself. Training a model to minimize expected loss — the probability-weighted average of how wrong its predictions are across the data distribution — is the formal objective behind nearly every supervised learning algorithm (Andrew Ng's CS229 lecture notes frame learning problems this way from the outset, defining a cost function whose expectation over the training distribution the algorithm seeks to minimize). Variance also underlies the bias-variance tradeoff central to model evaluation: a model with high variance fits its training data closely but produces wildly different predictions on new samples, while a model with high bias is consistently wrong in the same direction regardless of which training sample it saw; conditional probability, meanwhile, is the seed from which the full machinery of Bayesian updating — covered in depth later in this curriculum — grows directly out of the ideas introduced in this lesson.