Bayesian Methods

Bayes' theorem is a single formula that tells us precisely how to update a belief in light of new evidence: P(H | E) = P(E | H) · P(H) / P(E), where H is a hypothesis and E is observed evidence (Bishop's Pattern Recognition and Machine Learning develops this as the foundation of the book's Bayesian viewpoint). Three named quantities appear in this formula: the prior, P(H), which is the probability assigned to the hypothesis before seeing any evidence; the likelihood, P(E | H), which is the probability of observing the evidence if the hypothesis were true; and the posterior, P(H | E), the updated probability of the hypothesis after the evidence has been taken into account. Bayesian inference is, in essence, the disciplined process of turning a prior belief plus new data into a posterior belief, replacing gut-feeling updates with an exact, repeatable calculation.

The disease-testing scenario in this module's essential question is a classic worked illustration of exactly why intuition often fails here. Let the prior probability of disease be P(D) = 0.01, the test's sensitivity (true positive rate) be P(+ | D) = 0.9, and its false positive rate be P(+ | not D) = 0.05. Bayes' theorem gives P(D | +) = [P(+ | D) × P(D)] / [P(+ | D) × P(D) + P(+ | not D) × P(not D)] = (0.9 × 0.01) / (0.9 × 0.01 + 0.05 × 0.99) = 0.009 / 0.0585 ≈ 0.154 — meaning a positive test result raises the probability of disease only to about 15.4%, not 90%, because the disease is rare enough that false positives among the large healthy population outnumber true positives (MIT 18.05's coverage of Bayesian updating develops this same base-rate reasoning, and it is one of the most consistently miscalculated problems in introductory statistics).

This same mechanism drives real AI systems. A naive Bayes spam classifier treats 'is this email spam' as the hypothesis and the words in the email as evidence, multiplying a prior probability of spam by the likelihood of observing each word under the spam and non-spam hypotheses to arrive at a posterior probability of spam given the full message (Russell & Norvig's AIMA presents naive Bayes as a canonical probabilistic classifier). More broadly, any AI system that must revise its beliefs as new data streams in — from a robot updating its estimate of an obstacle's position after each sensor reading to a recommender system updating a user's inferred preferences after each click — is running the same prior-times-likelihood-equals-posterior computation introduced here, over and over, at every new piece of evidence it observes, with yesterday's posterior serving as tomorrow's prior.

Once a posterior distribution over a hypothesis or parameter has been computed, an AI system still has to act — and Bayesian decision theory provides the rule for doing so rationally: choose the action that minimizes expected loss under the posterior (Bishop's PRML develops this formally, connecting probability theory directly to decision-making). A common simplification is the maximum a posteriori (MAP) estimate, which simply picks the single hypothesis or parameter value with the highest posterior probability, ignoring the rest of the distribution's shape; in the disease-testing example, if 15.4% posterior probability of disease was still the most likely single diagnosis among the options considered, MAP would select it, even though the full posterior — carrying the actual 15.4% figure — contains far more usable information than that single point estimate does on its own, since a downstream decision-maker may need the exact probability, not just the most likely label, to weigh the cost of a missed diagnosis.

Working with the full posterior rather than collapsing it to a single point matters most when predicting new, unseen data. The posterior predictive distribution combines uncertainty about model parameters (captured in the posterior) with the model's own randomness to produce a prediction that honestly reflects both sources of uncertainty, rather than pretending the parameters are known exactly (Bishop's PRML; CS229's notes similarly motivate why point estimates like maximum likelihood can understate true predictive uncertainty). This is precisely the idea behind Bayesian neural networks and other uncertainty-aware AI models: instead of learning one fixed set of weights, they maintain a posterior distribution over plausible weight settings, and their predictions are computed by averaging over that whole distribution, which is why they can report not just a prediction but a calibrated measure of how confident they genuinely are in it, a property that matters enormously whenever an AI system's mistakes carry real-world consequences.

The contrast with purely frequentist statistical methods, covered in the previous module, is instructive rather than adversarial: frequentist inference treats a parameter as fixed but unknown and asks how an estimator behaves across repeated sampling, while the Bayesian approach treats the parameter itself as a random variable and directly computes a probability distribution over it given the data actually observed (AIMA discusses this as one dimension along which probabilistic AI approaches differ). In practice, AI systems often blend both traditions freely — a naive Bayes classifier's word-probability parameters might be estimated by maximum likelihood, a frequentist technique, and then combined via Bayes' theorem to produce Bayesian posterior predictions, illustrating that the two frameworks introduced across these two modules are complementary tools in the same statistical toolbox, not mutually exclusive philosophies to choose between once and for all, and that a working AI practitioner benefits from being fluent in both.