Advanced Reinforcement Learning

Every method covered so far in this course's reinforcement learning modules is model-free: Q-learning, policy gradients, and actor-critic all learn directly from trial-and-error experience, without ever explicitly representing how the environment's states transition or what rewards it produces. Model-based reinforcement learning instead learns, or is given, an explicit model of that transition and reward dynamics, and uses it to plan or to generate additional simulated experience. Sutton's 1991 Dyna architecture is the foundational example: it interleaves real experience from the environment with simulated experience sampled from a learned model, using both to update the same value function or policy, so that computation spent "imagining" transitions substitutes for some of the real interaction an agent would otherwise need. This matters most when real interaction is expensive, slow, or risky — a robot's joints wear out, and a poorly chosen real-world action can cause damage — so trading some real environment steps for cheap simulated ones can be a substantial sample-efficiency win, at the cost of relying on a model that may itself be imperfect.

Offline reinforcement learning, sometimes called batch RL, pushes this idea to its limit: the agent may only learn from a fixed, previously collected dataset of transitions, with no further interaction with the environment allowed at all. Levine, Kumar, Tucker, and Fu's 2020 tutorial and review identifies the central obstacle as distributional shift, also called extrapolation error: a value-based method like Q-learning bootstraps its target using max_a' Q(s',a'), which requires evaluating the learned Q-function at actions the current policy might choose — but if those actions are poorly represented or entirely absent from the fixed dataset, the Q-function's estimate there is unconstrained by any real data and can be wildly, systematically overestimated, and this error compounds across bootstrapped updates rather than averaging out. Because naive off-policy algorithms trained this way are prone to diverge or produce policies that look good on paper but fail catastrophically when deployed, offline RL methods generally constrain the learned policy to stay close to the data-generating behavior policy, or explicitly penalize actions the dataset does not support. Levine et al. are explicit that this remains an active, unsolved research area rather than a settled one — offline RL is exactly the setting a hospital with only historical treatment records, and no ability to safely experiment further, would need.

Long-horizon tasks with sparse rewards are notoriously hard for flat, single-level RL: if a meaningful reward only arrives after dozens or hundreds of correct actions in sequence, an agent exploring randomly is extremely unlikely to ever stumble into it. Hierarchical reinforcement learning addresses this by decomposing a task into a hierarchy of sub-policies operating at different timescales. Kulkarni, Narasimhan, Saeedi, and Tenenbaum's 2016 h-DQN is a canonical example: a higher-level meta-controller selects a subgoal (such as "reach a particular object"), and a lower-level controller learns a policy to achieve that specific subgoal, receiving an intrinsic reward whenever it succeeds, independent of whatever sparse extrinsic reward the overall environment eventually provides. This lets exploration happen at the more tractable level of "how do I reach this one subgoal" rather than "how do I stumble into the entire long sequence that eventually pays off," which the authors show helps substantially on sparse-reward domains that defeat flat Q-learning.

Imitation learning takes a different route around the exploration problem entirely: rather than learning from a reward signal, the agent learns to reproduce behavior demonstrated by an expert. The simplest version, behavior cloning, treats this as ordinary supervised learning — map observed states to the expert's actions — but suffers from compounding errors, since once the learned policy drifts even slightly off the expert's state distribution, it has no training signal telling it how to recover, and small mistakes snowball. Ho and Ermon's 2016 Generative Adversarial Imitation Learning (GAIL) reframes imitation as distribution matching instead: a discriminator network is trained to distinguish expert trajectories from trajectories generated by the current policy, while the policy is trained, via policy-gradient reinforcement learning, to fool that discriminator — directly matching the distribution of behavior rather than only mimicking individual state-to-action mappings, and sidestepping the need to hand-design a reward function.

Inverse reinforcement learning (IRL) asks a related but different question: rather than copying an expert's actions, can the reward function the expert appears to be optimizing be recovered? Ng and Russell's 2000 paper formalized this as inferring a reward function under which the expert's observed, assumed-near-optimal behavior is in fact optimal, given the environment's known dynamics. This is valuable precisely because a recovered reward function is more portable than a cloned policy: the same inferred reward can, in principle, be handed to any RL algorithm to derive a policy, or transferred to a related task with different dynamics, in a way that directly copying an expert's actions cannot be. Ng and Russell are careful to note that IRL is fundamentally ill-posed, however — many different reward functions, including a trivially uniform one, can rationalize the exact same observed behavior — so recovering a useful reward requires additional assumptions or regularization beyond the raw demonstrations.

Multi-agent reinforcement learning studies settings where several learning agents interact in a shared environment, each of whose actions affect the others' observations and rewards. This breaks a core assumption behind the convergence guarantees of Q-learning and other single-agent methods: from any one agent's perspective, the environment appears non-stationary, because the other agents are simultaneously updating their own policies, so the same state and action can lead to different outcomes over time even though the physical environment itself has not changed. Lowe, Wu, Tamar, Harb, Abbeel, and Mordatch's 2017 MADDPG addresses this with centralized training and decentralized execution: during training, each agent's critic is given access to the observations and actions of every agent, which makes the environment effectively stationary again from that critic's point of view, while each agent's actor is restricted to using only its own local observation at execution time, so the resulting policies can still act independently once deployed. For illustration, consider two agents each with a 4-dimensional local observation and a 2-dimensional action: a centralized critic trained on both agents' information would take a 4+4+2+2 = 12-dimensional input, while each agent's decentralized actor at execution time uses only its own 4-dimensional observation — the asymmetry between training-time and execution-time information is the whole trick.

Safe reinforcement learning is concerned with agents that must respect constraints, not merely maximize expected reward, which becomes essential as RL moves from games and simulators into settings — robotics, healthcare, infrastructure — where mistakes have real, sometimes irreversible costs. Amodei, Olah, Steinhardt, Christiano, Schulman, and Mané's 2016 "Concrete Problems in AI Safety" frames five practical, empirically approachable research problems relevant to modern RL systems rather than speculative long-term concerns: avoiding negative side effects an agent's reward function did not anticipate; avoiding reward hacking, where an agent exploits a loophole in its reward specification rather than doing what was intended; scalable oversight, providing enough supervisory feedback without requiring a human to check every action; safe exploration, so that trying new actions to learn does not itself cause catastrophic harm; and robustness to distributional shift, so a policy trained in one setting degrades gracefully rather than catastrophically when the environment it faces later differs from training.

Model-based and offline methods, hierarchical decomposition, imitation and inverse RL, multi-agent learning, and safety constraints are not separate, competing fixes so much as different responses to the same underlying observation: the flat, single-agent, purely online, reward-only MDP this course began with is a simplification, and each of these subfields relaxes exactly one of its assumptions. In practice, real deployments often need several relaxations at once — a fleet of coordinating multi-agent robots trained partly from offline logs still needs safe-exploration guarantees before any residual online fine-tuning is allowed, and a hierarchical controller learned via imitation still needs its subgoals checked against the same negative-side-effects and reward-hacking concerns Amodei et al. raise for flat policies. All of these — offline RL's distributional-shift problem, IRL's ill-posedness, multi-agent non-stationarity, and the five safe-RL problems above — remain active areas of ongoing research rather than settled engineering practice; this is precisely where much of the current frontier of reinforcement learning sits, well beyond the single-agent MDP this course opened with.