Simulation-aided Reinforcement Learning with Control Variates
Simulation-aided Reinforcement Learning with Control Variates
This paper introduces a conservative simulation-aided reinforcement learning method based on control variates. Unlike standard approaches that directly mix simulated and real data—which can degrade performance due to simulator inaccuracies—this method uses simulated data solely to reduce variance in the policy evaluation process, a fundamental building block of RL algorithms. The approach is integrated into the actor-critic framework for policy improvement and extended to offline RL scenarios. Experiments on multiple MuJoCo benchmark tasks show that this method outperforms standard simulation mixing approaches, highlighting control variates as an effective tool for improving simulation-aided reinforcement learning.
The authors propose a conservative simulation-aided reinforcement learning method using control variates to reduce variance without directly shaping the learning target, demonstrating improvements on MuJoCo benchmarks compared to standard simulation mixing approaches.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Environment
- SimulationOpen source S2
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
arXiv:2611.12643v1 Announce Type: cross Abstract: Reinforcement Learning (RL) algorithms often rely on data collected from a target environment, which can be expensive and noisy. Although simulators can generate large amounts of additional data, standard simulation mixing approaches, which directly combine simulated and real experience, can degrade performance due to simulator inaccuracies, especially when they bias
Open source S2
Source：arXiv Robotics — research abstracts · arxiv.org