Robot Learning and Human–Robot Interaction

The classical robotics pipeline separates perception, which turns raw sensor data into an estimate of the world, from control, which turns that estimate into motor commands, with each stage engineered and tuned largely independently. Levine, Finn, Darrell, and Abbeel's 2016 paper "End-to-End Training of Deep Visuomotor Policies" tests whether jointly training these stages together, rather than separately, produces better manipulation behavior, by training a single deep convolutional neural network policy, containing roughly 92,000 parameters, to map raw camera images directly to torques at the robot's motors. Rather than trying to solve this very high-dimensional policy-search problem directly, they use guided policy search, which converts it into a supervised learning problem: a separate trajectory-centric reinforcement learning process first generates good example trajectories under partial state information, and the deep visuomotor policy is then trained to imitate the actions those guiding trajectories took, conditioned only on the raw image and robot state it would actually have access to at test time. Evaluated on real-world manipulation tasks that demand tight coordination between what the robot sees and how it moves, including screwing a cap onto a bottle, this joint training approach demonstrated that perception and control can benefit from being learned together end-to-end, a genuinely different lesson from the strict separation between sensing, planning, and acting that opened this course's earlier robotics module.

A different way to acquire a manipulation skill is to learn directly from a demonstration rather than through trial-and-error reinforcement learning, an approach broadly called imitation learning. Duan, Andrychowicz, Stadie, Ho, Schneider, Sutskever, Abbeel, and Zaremba's 2017 paper "One-Shot Imitation Learning" pushes this further with a meta-learning framework: instead of training a policy on many demonstrations of one exact task, they train a neural network across a large distribution of related tasks so that, at test time, showing it just one new demonstration is sufficient for it to reproduce that behavior on a new instance of the task it has never encountered, with no additional training. The network takes both a single demonstration and the robot's current state as input and, using soft attention over the demonstration, outputs actions intended to reproduce the demonstrated behavior in the new situation; the authors evaluate this on block-manipulation tasks such as stacking every block on a table into a single tower, or arranging them into separate two-block towers, with block counts and starting arrangements varied at test time. This differs fundamentally both from classical reinforcement learning, which typically needs a hand-designed reward function and many trials per new task, and from ordinary imitation learning, which typically needs many demonstrations of the exact task being learned: one-shot imitation learning instead learns a general strategy for turning a single demonstration into a working policy.

Taken together, jointly trained visuomotor policies and one-shot imitation learning illustrate that "learning a manipulation skill" is not one single recipe but a family of different trade-offs. Reinforcement learning, whether trained on the real robot or, as in this course's earlier sim-to-real coverage, entirely in simulation, requires no human demonstrations at all but typically needs a well-specified reward signal and a large number of trials. Guided policy search narrows that trial requirement by bootstrapping from trajectory-centric optimization under favorable conditions and then training a general, sensor-driven policy to reproduce it. One-shot imitation learning goes furthest toward minimizing what a person must provide for each new task, a single demonstration, by shifting nearly all of the learning effort into an offline meta-training phase across many related tasks in advance. None of these approaches replaces the others; which one is appropriate for a given manipulation problem depends on how expensive real-world trials are, how many related tasks are available to learn from in advance, and how precisely a reward function for the task can actually be written down.

This course's earlier robotics module covered how OpenAI trained a policy in simulation to reorient a block in a Shadow Dexterous Hand using domain randomization, and how a follow-up system extended that approach to solving a physical Rubik's cube using automatic domain randomization, or ADR, to widen the simulated randomization ranges as training progressed. What that earlier coverage left unexamined is exactly how the "automatic" part of ADR works mechanically, described in OpenAI's 2019 paper "Solving Rubik's Cube with a Robot Hand": training begins in a fixed, non-randomized simulated environment, and whenever the current policy's performance under the current randomization ranges, over parameters such as the cube's simulated size, surface friction, and visual materials, exceeds a set performance threshold, those ranges are automatically widened, making the task modestly harder; training then continues under the new, wider ranges, and this cycle of succeed-then-widen repeats indefinitely. The degree of randomization the policy experiences at any point in training is therefore set automatically by how well it is currently performing, rather than by a fixed schedule an engineer chose in advance, which is what let training difficulty keep escalating without requiring continual manual re-tuning.

Beyond the headline scramble-solving results already covered, the researchers report a further, more qualitative finding: when the resulting policy was evaluated on the physical robot, it proved robust to real-world disturbances that were never modeled anywhere in its simulated training distribution at all, for instance continuing to solve the cube while being physically prodded with a stuffed toy giraffe. The researchers present this as evidence that sufficiently broad automatic domain randomization can produce generalization to genuinely novel, out-of-distribution perturbations, not merely robustness within the specific ranges of friction, mass, or appearance that were explicitly randomized during training. This is an honest, bounded, empirical observation on one hardware platform and one task family, not a general guarantee that domain randomization always produces this kind of out-of-distribution robustness, and understanding when and why it does remains an open question in sim-to-real research.

Peng, Andrychowicz, Zaremba, and Abbeel's 2018 paper "Sim-to-Real Transfer of Robotic Control with Dynamics Randomization" demonstrates that the same underlying strategy generalizes well beyond one dexterous hand and one manipulation task. Rather than randomizing primarily visual appearance and surface friction for an in-hand manipulation task, they randomize the simulator's dynamics parameters directly, including mass, friction, and motor strength, and apply this to a robotic arm performing an object-pushing task, a different embodiment and a different skill entirely from the earlier dexterous-hand research. They report that policies trained under this dynamics randomization maintained a similar level of performance when deployed on the real robot and were robust to substantial calibration error between the simulator's assumed dynamics and the real robot's actual dynamics. Where the dexterous-hand work showed domain randomization succeeding for one hand and one task, dynamics randomization shows the same core idea, training under many simulated variations so that no single simulated instance is over-fit to, generalizing across different physical quantities, different robot embodiments, and different tasks, evidence that domain randomization is a general strategy for narrowing the reality gap rather than an ad hoc trick specific to any one piece of hardware.

Goodrich and Schultz's 2007 survey "Human-Robot Interaction: A Survey" defines human-robot interaction, or HRI, as the field dedicated to understanding, designing, and evaluating robotic systems for use by or with humans, and emphasizes that meaningful interaction requires communication whether the robot and the person are physically co-located or working remotely from each other. They organize the field's research around five dimensions a designer can shape: the level of autonomy, ranging from full teleoperation, in which a human directly controls every robot action, to full autonomy, with many intermediate mixed-initiative points where control shifts between human and robot depending on the situation; the nature of information exchange, meaning what channels, such as visual displays, gesture, speech, or touch, and what content pass between human and robot; team structure, meaning how many humans and robots are involved and how authority and roles are distributed among them; adaptation, learning, and training, meaning how the robot's behavior, and the human operator's skill, change over repeated interaction; and task-shaping, meaning how introducing a robot changes the way the surrounding human work itself gets organized. This taxonomy gives researchers a common vocabulary for comparing HRI systems that otherwise look completely different, from a teleoperated bomb-disposal robot to a fully autonomous warehouse arm.

Lasota, Fong, and Shah's 2017 survey "A Survey of Methods for Safe Human-Robot Interaction" makes a point that reframes what "safety" even means in this context: they define safety to include "all possible ways in which harm could come to a person, ranging from physical contact to adverse psychological effects," deliberately broader than collision avoidance alone. They organize the resulting research into four categories of safety strategy. Safety through control regulates the robot's own motions and forces in real time, for example limiting the speed or force with which it can make contact with a person. Safety through motion planning chooses trajectories in advance that keep the robot's planned path away from a person, or away from a region a person is predicted to occupy, directly reusing the path-planning machinery from the previous lesson, but now treating a person as a special kind of obstacle a planner must route around. Safety through prediction anticipates where a person is likely to move next and adjusts robot behavior preemptively rather than only reacting once a person is already too close. Safety through consideration of psychological factors attends to a person's comfort and trust, since a robot can be technically safe from causing injury and still behave in ways that startle or unsettle the people working near it.

These academic safety strategies connect directly to deployed industrial practice through ISO/TS 15066, a real technical specification governing collaborative robots, or "cobots," that share a workspace with people without a safety cage separating them. Among the collaboration modes it defines are power- and force-limiting, which restricts the force and pressure a robot can exert during incidental contact with a person to below biomechanically established injury thresholds so that a collision that does occur causes no lasting harm, and safety-rated monitored stop, which halts the robot's motion whenever a person enters the shared workspace and resumes automatically once the person has left, a direct industrial instance of Lasota, Fong, and Shah's "safety through control" category. Learned robot behaviors, whether from the demonstration-based and jointly-trained policies in this module's earlier lessons or the simulation-trained, randomized policies of this course's sim-to-real coverage, raise a safety question that classical, hand-engineered systems do not have to answer in the same way: how do you certify that a policy trained from data and simulation, whose behavior in genuinely novel real-world situations cannot be exhaustively enumerated in advance, will still respect these same safety guarantees, an active area of ongoing robotics and machine-learning-safety research rather than a solved problem.