Goodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences
Goodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences
This research investigates how sampling-based control methods, such as model-predictive control and best-of-N selection, exploit reward models learned from human preferences. Using benchmarks like HalfCheetah, Hopper, Meta-World drawer-open, and plate-slide, the authors show that harder MPPI searches can reduce true return by up to 76% when using human or synthetic labels. They prove that chi-squared regularized MPPI with temperature set by the scorer's error spread is worst-case optimal and propose a calibration method from 100 proposal pairs ranked by an oracle, which retains 86-94% of the cloned policy's return on specific tasks while mitigating exploitation.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
proposal pairs ranked by true return
100
View original evidence
vers the selection terms. Calibrated from 100 proposal pairs ranked by true return (an oracle) on start states the evaluation never uses, this rule is the safest fix we test: with each of five independent sets it succeeds in every arm run and keeps on average 86-94% of the cloned policy's return on plate-slide and drawer-open, though it is cautious where search helps. Calibration from a person's comparisons cannot de
Open source S5
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
MPPI search cuts true return by up to 76%: with human labels the drawer opens in 0 of 20 runs against 10 of 10 for the cloned policy the planner starts from, while held-out label accuracy stays at 0.75-0.97. On plate-slide human labels are exploited less than synthetic ones, and on window-open search helps. True return roughly follows how far the planner's weights move from its proposals ($\sqrt{\mathrm{KL}}$) for b
Open source S3
vers the selection terms. Calibrated from 100 proposal pairs ranked by true return (an oracle) on start states the evaluation never uses, this rule is the safest fix we test: with each of five independent sets it succeeds in every arm run and keeps on average 86-94% of the cloned policy's return on plate-slide and drawer-open, though it is cautious where search helps. Calibration from a person's comparisons cannot de
Open source S5
Source：arXiv Robotics — research abstracts · arxiv.org