Abstract
Specifying reward functions for robot manipulation is notoriously difficult. Hand-coded rewards that look correct in simulation routinely fail in deployment because they inadvertently reward proxy behaviors that diverge from the intended goal. Learning reward functions from human feedback sidesteps the specification problem but depends on the feedback modality. Corrective interventions - physical takeovers or joystick corrections during task execution - are a high-bandwidth feedback channel that is natural for human supervisors and carries rich implicit information about what the robot should have done differently. Extracting structured reward signal from this implicit information is an active research area with direct relevance to deploying robots in human-occupied environments.
What Corrections Contain
When a human intervenes to correct a robot, the intervention trajectory implicitly encodes several things: that the pre-intervention robot behavior was suboptimal at the moment of takeover, that the corrective trajectory is closer to optimal, and sometimes (depending on the correction’s structure) a local direction of improvement in behavior space. Inverse reinforcement learning (IRL) algorithms applied to correction data can recover reward functions that rationalize the human’s choices, but standard IRL assumes corrections are globally optimal, which is rarely true: humans intervene when they notice a problem, not necessarily when the divergence is largest, and their corrective trajectories are locally optimal, not globally replanned from scratch. Methods that account for this bounded-rationality structure, including DAgger variants and COACH (Corrective Advice Communicated by Humans), outperform naive IRL on few-correction learning problems.
Physical Human-Robot Interaction as Feedback
Physically correcting a robot arm during a manipulation task requires the robot to be compliant, which creates a practical tension: high compliance makes the robot easy to guide but reduces task-execution stiffness. Admittance control frameworks that switch between task-execution and correction-acceptance modes based on applied force magnitude have been used in work from Stanford’s ILIAD lab and from the HRI group at Georgia Tech. The key design choice is the threshold: too low and the robot interprets incidental contact as correction; too high and gentle repositioning corrections are ignored. Learning this threshold from historical correction distributions is one area where data from deployed systems substantially outperforms manually tuned parameters.
Reward Function Uncertainty and Ambiguity
A single correction is often ambiguous: the corrective trajectory is consistent with many reward functions. Maintaining a posterior over reward functions rather than committing to a point estimate allows the robot to act cautiously in regions where the reward is uncertain and to query for additional corrections in maximally informative states. This Bayesian IRL framing, developed by Pieter Abbeel and Stuart Russell’s groups at Berkeley, has been applied to corrective intervention settings by teams at CHAI lab and extended to handle temporally extended corrections. The computational cost of posterior maintenance scales with the complexity of the reward function class; linear reward functions admit exact updates while neural reward models require approximate inference via ensembles or MCMC.
Dataset and Evaluation Gaps
The field lacks large-scale public datasets of corrective interventions from non-expert supervisors on varied manipulation tasks. Most published results use laboratory studies with small numbers of participants, often robotics-familiar. Datasets collected from human operators in real logistics or assembly environments would be substantially more useful for evaluating generalization. The REBUS dataset from ETH Zurich and the InterACT dataset from CMU are partial steps in this direction but remain limited in task diversity and operator demographics.