TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing
TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing
This paper introduces TOPPO (Tail-Optimized PPO), a method that reformulates Proximal Policy Optimization (PPO) to address critic-side gradient ill-conditioning in multi-task reinforcement learning. By implementing critic balancing modules, TOPPO improves gradient conditioning and balances learning dynamics across tasks. Empirical results on the Meta-World+ MT50 benchmark show that TOPPO achieves stronger mean and tail-task performance compared to prior SAC-family and ARS-family methods while using fewer parameters and reaching final performance in fewer environment steps. Ablation studies confirm the effectiveness of each module, demonstrating that with proper optimization, on-policy methods can rival or exceed off-policy approaches in multi-task settings.
The authors identify critic-side gradient ill-conditioning as a bottleneck in on-policy multi-task reinforcement learning and propose TOPPO, a reformulation of PPO using critic balancing modules to improve tail-task performance.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
arXiv:2605.11473v2 Announce Type: replace-cross Abstract: Soft Actor-Critic (SAC) and its variants dominate Multi-Task Reinforcement Learning (MTRL) due to their off-policy sample efficiency, while on-policy methods such as Proximal Policy Optimization (PPO) remain underexplored. We diagnose that PPO in MTRL suffers from a previously overlooked issue: critic-side gradient ill-conditioning, which may cause tail tasks
Open source S2
to stall while easy tasks dominate the value function's updates. To address this, we propose TOPPO (Tail-Optimized PPO), a reformulation of PPO via Critic Balancing -- a set of modules that improve gradient conditioning and balance learning dynamics across tasks. Unlike prior approaches that rely on modular architectures or large models, TOPPO targets the optimization bottleneck within PPO itself. Empirically, on the
Open source S3
Meta-World+ MT50 benchmark, TOPPO achieves stronger mean and tail-task performance than prior SAC-family and ARS-family methods while using substantially fewer parameters and reaching their final performance in fewer environment steps. Notably, TOPPO matches or surpasses strong SAC baselines early in training and maintains superior performance at full budget. Ablations confirm the effectiveness of each module in TOP
Open source S4
PO and provide insights into their interactions. Our results demonstrate that, with proper optimization, on-policy methods can rival or exceed off-policy approaches in MTRL, challenging the prevailing reliance on SAC and highlighting critic-side gradient conditioning as the central bottleneck.
Open source S5
TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing
Open source S1
Source：arXiv Robotics — research abstracts · arxiv.org