AI Agents

In their widely used textbook Artificial Intelligence: A Modern Approach, Stuart Russell and Peter Norvig define an agent as anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators, and they organize agent programs into a hierarchy of four increasingly general designs. The simplest is the simple reflex agent, which selects actions based only on the current percept, ignoring the rest of the percept history entirely; these agents are built around condition-action rules of the form 'if condition then action,' such as a rule that says if the car in front is braking, then initiate braking. Because a simple reflex agent has no memory of anything beyond what it senses right now, Russell and Norvig note that its rationality depends on the environment being fully observable at each moment, and that such agents will often fail or loop indefinitely in environments where the correct action depends on history rather than the present percept alone — a classic textbook example is a reflex vacuum-cleaner agent that cleans a square if it is dirty and otherwise moves to the next square. The next step up the hierarchy is the model-based reflex agent, which copes with partial observability by maintaining an internal state that tracks aspects of the world not revealed in the current percept; this internal state is updated over time using information about how the world evolves independently of the agent and how the agent's own actions affect the world, which Russell and Norvig describe as an update-state function taking the previous state, the most recent action, and the new percept, and returning an updated description of the current world state. A model-based reflex agent still ultimately selects its action through condition-action rules, exactly as a simple reflex agent does, but the conditions those rules test can now refer to the maintained internal state rather than only to the instantaneous percept, which allows the agent to behave sensibly in environments where the same percept can call for different actions depending on what has happened before — the difference between a vacuum-cleaning agent that remembers which squares it has already cleaned and one that must re-sense every square from scratch on every time step.

Beyond reacting to state, an agent often needs to know what it is trying to achieve, which motivates the goal-based agent. In this design the agent combines its model of how the world works with an explicit representation of a goal, a desired configuration of the world, and then uses search or planning to decide which sequences of actions will lead from the current state to a state satisfying that goal. Because goal-based agents reason about the future consequences of their actions rather than simply reacting to the present, Russell and Norvig characterize this kind of decision-making as more flexible than reflex-based approaches, since the same underlying knowledge about the environment can be reused to pursue different goals simply by changing what goal is supplied to the agent. Course notes covering this material describe the goal-based agent as strictly more general than the model-based reflex agent because it retains the model of the environment while adding this goal abstraction, allowing the agent's behavior to generalize across problems that share the same environment dynamics but differ in what outcome is desired. The tradeoff is that goal-based agents require more computation, since evaluating whether a hypothetical sequence of actions will actually achieve the goal typically involves searching over possible future states rather than simply looking up a rule.

The most general design in the classical hierarchy is the utility-based agent, which is needed because goals alone are a fairly coarse way of distinguishing good states from bad ones. Goals provide only a binary distinction between success and failure, but many real environments involve conflicting goals, only some of which can be achieved, or goals that can only be reached with some degree of likelihood, and in these cases an agent needs a way to weigh tradeoffs. A utility-based agent addresses this by using a utility function, which maps a single state, or a sequence of states, onto a real number that represents how desirable that outcome is, so that the agent can compare and rank world states rather than simply asking whether a goal has been met. Russell and Norvig describe the utility function as effectively an internalization of the agent's performance measure, and note that a rational utility-based agent chooses the action that maximizes its expected utility, which becomes essential once uncertainty is introduced into the environment because the agent must weigh the probabilities of different outcomes against their utilities. This utility-maximizing formulation subsumes the goal-based case, since a goal can be represented as a utility function that assigns high utility to goal states and low utility elsewhere, while also supporting the more nuanced reasoning needed when several competing objectives must be balanced against each other.

The classical taxonomy from Russell and Norvig describes the internal architecture an agent program can use to map percepts to actions, but it says little about what kind of reasoning process actually generates those decisions inside a goal-based or utility-based agent. Over the last several years, researchers have begun using large language models themselves as the reasoning component of an agent, prompting or fine-tuning them to decide which actions to take, observe the results, and decide what to do next, effectively turning a language model into the decision-making core of a goal-directed system. A foundational paper in this line of work is Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao's 'ReAct: Synergizing Reasoning and Acting in Language Models,' which proposes prompting large language models to generate both verbal reasoning traces, referred to as 'thoughts,' and task-specific actions in an interleaved manner, rather than generating one type of output alone. This interleaving is the paper's central technical contribution, and it distinguishes ReAct-style agents from earlier approaches that either asked a language model to reason in free text without ever taking actions in an environment, or asked it to select actions directly without any interposed verbal reasoning step.

The ReAct paper argues that reasoning and acting are complementary and each helps the other. Generated reasoning traces let the model induce, track, and update action plans, and even handle exceptions or unexpected observations, because the model can explicitly think through what it has learned so far before deciding what to do next, while the ability to act lets the model interface with external sources of information, such as a search API or a knowledge base, so that its reasoning can be grounded in up-to-date, task-specific evidence rather than relying purely on what was memorized during training. In the ReAct loop, at each step the model produces a thought describing its current reasoning about the task, then produces an action, such as a query to a Wikipedia search API in the paper's question-answering experiments, then receives an observation returned by that action, and this thought-action-observation cycle repeats until the model determines it has enough information to produce a final answer. Yao et al. show that on knowledge-intensive question-answering and fact-verification benchmarks this approach reduces the hallucination and error-propagation problems that plague reasoning-only prompting, because incorrect intermediate reasoning can be corrected by new observations from the environment, and on interactive decision-making benchmarks ReAct outperformed action-only baselines by large absolute margins while requiring only one or two in-context examples to prompt effectively.

Mapping this modern approach back onto Russell and Norvig's classical taxonomy clarifies both what has changed and what has stayed constant. A ReAct-style language model agent is still, at the architectural level Russell and Norvig describe, best understood as a goal-based or utility-based agent, since it is given a task or objective and must choose a sequence of actions intended to satisfy it, and it typically must track state across the interaction, much like a model-based reflex agent tracks state, since each new observation needs to be interpreted in light of everything gathered so far. What is genuinely new is not the high-level control architecture but the mechanism used to implement the decision-making inside that architecture: instead of hand-written condition-action rules, explicit search over a symbolic state space, or a manually engineered utility function, a ReAct agent uses a large language model's learned, natural-language reasoning ability, expressed as free-text thoughts, to implicitly perform the planning and evaluation that the classical agent types formalize more explicitly. This makes modern LLM agents considerably more flexible in open-ended, language-rich domains than hand-coded classical agents, since the same pretrained model can be redirected to a new goal simply by changing the prompt or task description rather than by rewriting rules or a state representation, but it also means their behavior is comparatively harder to formally verify or guarantee correct than the more structured classical designs, which is an active area of ongoing research.