Semantics-to-Contact: A Stagewise Framework for Robust Contact-Rich Manipulation
Semantics-to-Contact: A Stagewise Framework for Robust Contact-Rich Manipulation
This research introduces Semantics-to-Contact (S2C), a stagewise framework designed to enhance visually robust contact-rich manipulation. The approach decomposes the problem into semantic focusing, where a vision-language model localizes task-relevant regions in diverse visual scenes, and contact refinement, where a human-in-the-loop residual reinforcement learning policy learns fine-grained behaviors using force feedback within those localized regions. To further boost visual robustness, an object-centric visual augmentation strategy randomizes background appearance while preserving the robot and relevant objects. Experiments across six real-world manipulation tasks demonstrate that S2C consistently outperforms strong baselines in success rates and maintains reliable performance under significant visual and positional variations.
The paper proposes a two-stage framework combining vision-language semantic localization with human-in-the-loop residual reinforcement learning to improve robustness in contact-rich manipulation tasks under visual variations.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Control
- Shared controlOpen source S3
- paper: not_reported
- code: not_reported
- weights: not_reported
- dataset: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
rce feedback. To further improve visual robustness, we introduce an object-centric visual augmentation strategy that randomizes background appearance while preserving the robot and task-relevant objects. Experiments on six contact-rich real-world manipulation tasks exhibit that S2C consistently improves success rates over strong baselines and maintains reliable performance under significant visual and positional vari
Open source S4
e the problem into two stages: semantic focusing and contact refinement. In the first stage, a vision-language model localizes task-relevant regions across visually diverse scenes, providing coarse spatial guidance and reducing the burden of exploration. In the second stage, a real-world human-in-the-loop residual reinforcement learning policy learns fine-grained contact behaviors within the localized region using fo
Open source S3
ations. These results demonstrate that stagewise semantic focusing and contact refinement provide a practical path toward visually robust real-world contact-rich manipulation. Our demos can be seen in https://edem-ai.github.io/S2C/.
Open source S5
Source：arXiv Robotics — research abstracts · arxiv.org