Decision Theory and Utility

Decision theory, as the Stanford Encyclopedia of Philosophy frames it, studies "the reasoning underlying an agent's choices," and its dominant normative framework — expected utility (EU) theory — holds that "in situations of uncertainty, one should prefer the option with greatest expected desirability or value". The framework starts from a preference relation over outcomes or "prospects," which is considered rational only if it satisfies certain consistency axioms: completeness (for any two options, the agent can say it prefers one, prefers the other, or is indifferent) and transitivity (if A is preferred to B, and B to C, then A must be preferred to C), the basic requirements ruling out circular or gap-ridden preferences. MIT's graduate microeconomic theory sequence adds two further axioms needed specifically for choices involving risk: continuity and the independence axiom, which requires that mixing two lotteries with a third in the same proportions must preserve their original preference ordering.

The payoff of these axioms is the von Neumann–Morgenstern expected utility theorem: any preference relation satisfying them can be represented by a utility function u over outcomes such that one lottery is preferred to another exactly when it has higher expected utility, U(p) = Σ p(c)·u(c) summed over each possible consequence c. A worked example shows why this matters: suppose a decision-maker's utility for money follows u(x) = √x, and they must choose between a certain $50 and a 50/50 gamble between $100 and $0. The gamble's expected monetary value is 0.5(100) + 0.5(0) = $50, identical to the certain amount — but its expected utility is 0.5·√100 + 0.5·√0 = 0.5(10) + 0.5(0) = 5, strictly less than the certain option's utility of √50 ≈ 7.07. A rational agent with this utility function prefers the sure $50 despite the equal expected payoff.

That gap between the two options in the example is risk aversion, made mathematically visible: because √x is a concave function, its curve bends below the straight line connecting any two points on it, so a weighted average of utilities at two outcomes is always less than the utility of the weighted-average outcome. A linear utility function instead describes a risk-neutral agent (indifferent between the gamble and its expected value), and a convex utility function describes a risk-seeking agent (who would prefer the gamble). This is not merely a philosopher's abstraction — it is the same framework Russell and Norvig's foundational AI textbook uses to define a "rational agent" as one that chooses actions maximizing expected utility given its beliefs, making the choice of utility function itself a core design decision for any AI system operating under uncertainty.

A decision-theoretic AI agent combines two ingredients: a probability distribution representing its beliefs about the world, and a utility function representing its preferences over outcomes, then chooses whichever available action maximizes expected utility, EU(a) = Σ P(outcome | a, evidence)·U(outcome), summed over every possible outcome of taking action a. A worked example: imagine an agent deciding whether to carry an umbrella, with P(rain) = 0.3. Suppose the agent assigns utility +5 to carrying an umbrella in the rain (mild hassle, stays dry), +3 to carrying one when it doesn't rain (hassle only), −10 to going without one in the rain (soaked), and +10 to going without one when it's dry (maximally convenient). Then EU(carry) = 0.3(5) + 0.7(3) = 1.5 + 2.1 = 3.6, while EU(no umbrella) = 0.3(−10) + 0.7(10) = −3 + 7 = 4.0 — so, given these specific utility values, the expected-utility-maximizing choice is to leave the umbrella behind, illustrating that the "obviously safe" choice is not always optimal once utilities, not just probabilities, are taken into account.

Berkeley's introductory AI course formalizes this reasoning with decision networks (also called influence diagrams): Bayesian networks extended with two new node types, decision nodes representing the agent's choice and a utility node representing the payoff, so that the network directly computes "the action which maximizes the expected utility given the evidence". Because the network already encodes the agent's probabilistic beliefs as a Bayes net, adding decision and utility nodes lets the same machinery used for probabilistic inference double as a planner: querying the network for the expected-utility-maximizing setting of the decision node is a direct extension of ordinary Bayesian inference. These networks also support computing the value of perfect information (VPI) — how much expected utility would improve if the agent could learn some currently-unknown variable before acting — and a key, non-obvious property is that VPI is always nonnegative (learning more can never hurt an expected-utility maximizer, since the agent can always simply ignore new information) yet nonadditive (the value of learning two facts together need not equal the sum of their individual values, because the facts may be redundant or synergistic).

This same expected-utility machinery reappears, often unlabeled, throughout machine learning: a "loss function" is simply negative utility, and "expected risk minimization" — the standard training objective for a supervised learning model — is expected utility maximization under a different name, minimizing the expected loss over the data distribution rather than maximizing expected payoff over an outcome distribution. Framing model training this way clarifies why the choice of loss function is not a mere implementation detail but a genuine value judgment: choosing mean-squared-error versus cross-entropy versus a custom asymmetric loss is exactly the same kind of choice as choosing a concave versus linear utility function in classical decision theory, since both encode how costly different kinds of mistakes are considered to be. A fraud-detection model, for instance, might use an asymmetric loss that penalizes a missed fraud case far more heavily than a false alarm — precisely the same design move as an agent adopting a highly risk-averse utility function over financial outcomes, and both are decisions about values, not just mathematics.