Learning When to Think: Adaptive Internal Computation for Reinforcement Learning
Learning When to Think: Adaptive Internal Computation for Reinforcement Learning
This paper presents Adaptive Internal Computation (AIC), an iterative neural policy that learns when to stop refining its latent representation and act. By jointly training action selection and stopping with auxiliary supervision from expected-depth penalties and intermediate value predictions, AIC outperforms state-independent schedules on tasks where variable computation is beneficial. On twelve benchmark tasks, it executes an average of 3.3 refinement steps (with a maximum of eight), achieving a 31--44% reduction in per-decision latency relative to a fixed eight-step policy on four control and robotics tasks. The method exceeds the best fixed-depth policy in mean reward on seven tasks but trails on four. Ablation studies indicate that adaptive stopping contributes most of the reward gain on PickCube and AntMaze, while value supervision is more critical for RareEvent.
The authors introduce Adaptive Internal Computation (AIC), an iterative policy that learns to allocate variable computation steps based on state, reporting a 31--44% reduction in per-decision latency and higher mean rewards on seven of twelve benchmark tasks compared to fixed-depth baselines.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
arXiv:2611.10822v1 Announce Type: cross Abstract: Iterative neural policies usually spend the same computation on every decision, although the benefit of additional processing varies across states. We introduce Adaptive Internal Computation (AIC), an iterative policy that learns when to stop refining its latent representation and act. In AIC, we train action selection and stopping jointly, an expected-depth penalty v
Open source S2
alues computation and intermediate value predictions provide auxiliary supervision. We show that when only some states benefit from further refinement, an ideal state-dependent allocator outperforms every state-independent stopping schedule at the same expected depth. With at most eight refinement steps per decision, AIC executes 3.3 steps on average across twelve benchmark tasks and lowers per-decision latency by 31
Open source S3
--44% relative to a fixed eight-step policy on four control and robotics tasks. Its mean reward exceeds the best fixed-depth policy on seven tasks and trails it on four. Ablating adaptive stopping and value supervision separately shows that stopping accounts for most of the reward gain on PickCube and AntMaze, where it also executes more refinement steps and that supervision accounts for most of it on RareEvent.
Open source S4
Source：arXiv Robotics — research abstracts · arxiv.org