Statistics and Statistical Inference

Statistical inference begins with estimation: using a limited sample of data to infer something about a larger, unobserved population (MIT OCW 18.650, Statistics for Applications). The simplest and most common estimator is the sample mean — the average of the observed data points, used as a best guess for the true population mean when the population itself is far too large or too costly to measure in full. Suppose five separate measurements of a model's response latency (in milliseconds) are recorded: 120, 115, 130, 125, and 110. The sample mean is (120 + 115 + 130 + 125 + 110) / 5 = 600 / 5 = 120 milliseconds — an estimate of the model's typical latency across all possible requests it might ever receive, not just these five, computed from a necessarily incomplete sample of them, and it would likely change slightly if a different five requests had been measured instead.

Because any sample is just one of many possible samples that could have been drawn, an estimator like the sample mean is itself a random variable with its own sampling distribution — the distribution of values it would take across repeated samples of the same size (MIT OCW 18.05). A central result underlying much of applied statistics is that, as the sample size grows, the sampling distribution of the sample mean tends to look like a normal (Gaussian) distribution centered on the true population mean, regardless of the shape of the original data — this tendency, known as the Central Limit Theorem, is what makes it possible to attach a principled, quantitative measure of uncertainty to an estimate computed from a finite sample, rather than treating that estimate as if it were exact, and it explains why larger evaluation sets consistently produce more trustworthy accuracy figures than small ones.

This distinction between a single estimate and the estimator's sampling distribution is exactly why an AI team cannot simply trust a single accuracy number computed on one test set. If the test set is itself a sample from a larger population of possible inputs the deployed model will encounter, then the measured 87% accuracy is a sample mean with sampling variability of its own — a different randomly drawn 200-example test set could easily have produced 84% or 90% purely by chance, even if the model's true accuracy on the full population were unchanged (Andrew Ng's CS229 notes and Russell & Norvig's AIMA both frame generalization performance as fundamentally a statistical estimation problem, not a fixed fact about a model). The tools in the next lesson — confidence intervals and hypothesis tests — exist precisely to quantify that uncertainty rigorously.

A confidence interval turns a single point estimate into a range of plausible values for the true population parameter, together with a stated confidence level (MIT OCW 18.650). For a sample proportion p̂, the standard error is approximately √(p̂(1 − p̂)/n), and a 95% confidence interval is roughly p̂ ± 1.96 × standard error. Applying this to the accuracy example above — 174 correct predictions out of 200 test examples gives p̂ = 0.87, with standard error √(0.87 × 0.13 / 200) ≈ 0.0238, so the 95% confidence interval is approximately 0.87 ± 1.96(0.0238) ≈ (0.82, 0.92). That fairly wide interval — from 82% to 92% — makes clear that a single 87% reading on only 200 examples carries substantial uncertainty about what the model's true accuracy actually is on the full population of inputs it will ever see.

Hypothesis testing formalizes the question of whether an observed difference, like the three-point accuracy gap between two models, reflects a genuine effect or could plausibly be explained by chance alone (MIT OCW 18.05). The procedure starts by assuming a null hypothesis — here, that the two models have equal true accuracy — and asks how likely it would be to observe a gap at least as large as the one measured if that null hypothesis were true; this probability is the p-value, and a conventionally small p-value (often below 0.05) is treated as evidence against the null. If the confidence intervals for the two models' accuracies overlap substantially, as they well might given the (0.82, 0.92) interval computed above, that is itself evidence the observed three-point gap could easily arise from sampling variability rather than a real capability difference, and a proper hypothesis test, or simply a larger test set, is needed before concluding the new model is genuinely better.

Regression extends estimation to relationships between variables: linear regression estimates the coefficients of a line (or hyperplane) that best predicts an outcome variable from one or more predictor variables, and — like any estimator — those coefficients come with their own sampling uncertainty (MIT 18.650; CS229's lecture notes develop linear regression as the canonical first supervised learning algorithm, fit by minimizing squared prediction error). If a regression model predicts validation accuracy from the logarithm of training-set size and estimates a slope of 0.02, that means each additional order of magnitude of training data is associated with roughly a two-percentage-point increase in accuracy, holding other factors fixed — precisely the kind of quantitative, uncertainty-aware relationship that statistical inference, not just calculus-based curve fitting alone, is built to characterize and communicate honestly, including how much that 0.02 estimate itself could plausibly vary if a new batch of experiments were run.