All-For-One: Distilling Visual Foundation Models to a Fast, Memory-Efficient Visuomotor Policy
All-For-One: Distilling Visual Foundation Models to a Fast, Memory-Efficient Visuomotor Policy
This paper introduces All-For-One (AFO), a two-stage distillation framework that leverages Visual Foundation Models (VFMs) as privileged information during training to create a lightweight, RGB-only student policy. By decoupling semantic representation learning from inference-time execution, AFO achieves over 30x fewer parameters than VFM-based encoders and 50x lower latency than its teacher in simulation benchmarks. Evaluated on Adroit and MetaWorld tasks in simulation as well as three real-world manipulation tasks, the method outperforms prior foundation-model-based approaches while addressing issues of high computational cost, temporal inconsistency, and distribution shift between static internet images and egocentric robot observations.
The authors propose a two-stage distillation framework that decouples semantic representation learning from policy execution to reduce computational cost and latency while maintaining performance on simulated and real-world manipulation tasks.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Environment
- SimulationOpen source S4
Reported numbers
parameters
>30
View original evidence
knowledge is distilled in-distribution into a lightweight RGB-only student policy. Evaluated on two Adroit tasks and five MetaWorld tasks in simulation, as well as three real-world manipulation tasks, AFO outperforms prior foundation-model-based approaches while using over 30x fewer parameters than VFM-based encoders and 50x lower latency than its own teacher in simulation benchmarks, providing an efficient pathway f
Open source S4latency
50
View original evidence
knowledge is distilled in-distribution into a lightweight RGB-only student policy. Evaluated on two Adroit tasks and five MetaWorld tasks in simulation, as well as three real-world manipulation tasks, AFO outperforms prior foundation-model-based approaches while using over 30x fewer parameters than VFM-based encoders and 50x lower latency than its own teacher in simulation benchmarks, providing an efficient pathway f
Open source S4
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
knowledge is distilled in-distribution into a lightweight RGB-only student policy. Evaluated on two Adroit tasks and five MetaWorld tasks in simulation, as well as three real-world manipulation tasks, AFO outperforms prior foundation-model-based approaches while using over 30x fewer parameters than VFM-based encoders and 50x lower latency than its own teacher in simulation benchmarks, providing an efficient pathway f
Open source S4
Source：arXiv Robotics — research abstracts · arxiv.org