Goodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences
Overview
This research investigates how sampling-based control methods, such as model-predictive control and best-of-N selection, exploit reward models learned from human preferences. Using benchmarks like HalfCheetah, Hopper, Meta-World drawer-open, and plate-slide, the authors show that harder MPPI searches can reduce true return by up to 76% when using human or synthetic labels. They prove that chi-squared regularized MPPI with temperature set by the scorer's error spread is worst-case optimal and propose a calibration method from 100 proposal pairs ranked by an oracle, which retains 86-94% of the cloned policy's return on specific tasks while mitigating exploitation.
Generated from attributed reports · Updated 2 days ago
Event evidence and corrections
0 attributed source owners. Ownership does not establish independent confirmation. Quantities are reported separately and are never added together.
Reported quantity · proposal pairs ranked by true return: 100 other · Basis not reported
“vers the selection terms. Calibrated from 100 proposal pairs ranked by true return (an oracle) on start states the evaluation never uses, this rule is the safest fix we test: with each of five independent sets it succeeds in every arm run and keeps on average 86-94% of the cloned policy's return on plate-slide and drawer-open, though it is cautious where search helps. Calibration from a person's comparisons cannot de”
Exact source · revision 1Source owner not reported
Report timeline
Follow attributed reports and material updates.
- arXiv Robotics — research abstractsSignalGoodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences
This research investigates how sampling-based control methods, such as model-predictive control and best-of-N selection, exploit reward models learned from human preferences. Using benchmarks like HalfCheetah, Hopper, Meta-World drawer-open, and plate-slide, the authors show that harder MPPI searches can reduce true return by up to 76% when using human or synthetic labels. They prove that chi-squared regularized MPPI with temperature set by the scorer's error spread is worst-case optimal and propose a calibration method from 100 proposal pairs ranked by an oracle, which retains 86-94% of the cloned policy's return on specific tasks while mitigating exploitation.
Event coverage history
Current attention 3·Peak within the comparable range 4（2026-11-10 11:00 UTC）·Change within the comparable range over 24 hours –
The trend compares participants observed continuously; its scope may be narrower than current attention totals. Hover or click to explore hourly observations, or use the left and right arrow keys.
Timezone · UTC
Article dates follow your selected timezone. Briefing editions use Hong Kong time (UTC+8).