Listen Before Act: Audio-Conditioned Robot Manipulation with ListenVLA
Listen Before Act: Audio-Conditioned Robot Manipulation with ListenVLA
This paper presents ListenVLA, an audio-conditioned VLA policy that improves robotic manipulation by integrating auditory cues into action generation. The method combines semantic audio embeddings with waveform-derived acoustic cues and introduces an audio-action binding objective to link auditory differences to demonstrated actions. Evaluated on a new benchmark, LIBERO-Audio, and real-world tasks, the approach demonstrates strong performance in sound-based manipulation.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
tasks
8
View original evidence
ction differences. The objective supervises short-horizon differences between flow-velocity predictions under different audio conditions, with shared pure-noise action inputs and fixed non-audio observations, alongside full-chunk action supervision. Furthermore, we develop LIBERO-Audio, an eight-task benchmark covering sound-based target selection, responses to auditory events, and physical interactions that reveal a
Open source S4
- code: Available Artifact link Open source S5
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
ction differences. The objective supervises short-horizon differences between flow-velocity predictions under different audio conditions, with shared pure-noise action inputs and fixed non-audio observations, alongside full-chunk action supervision. Furthermore, we develop LIBERO-Audio, an eight-task benchmark covering sound-based target selection, responses to auditory events, and physical interactions that reveal a
Open source S4
dio-conditioned VLA policy built on GR00T N1.7 for sound-guided manipulation. ListenVLA combines semantic audio embeddings with waveform-derived acoustic cues over a causal audio history and conditions continuous action generation through audio tokens and feature modulation. Moreover, we introduce an audio-action binding objective that uses context-matched demonstrations to link auditory differences to demonstrated a
Open source S3
coustic evidence. Experiments on LIBERO-Audio and three real-world manipulation tasks show excellent performance. Ablation studies support the contributions of binding supervision and audio modulation, while audio swap tests show that sound influences action predictions. Our code and videos are available at https://anonymous.4open.science/r/listenvla-F18D
Open source S5
Source：arXiv Robotics — research abstracts · arxiv.org