PhysMoE: Fast Spatial Grounding with Single-Token Coordinate Representation in Modality-Decoupled VLMs
PhysMoE: Fast Spatial Grounding with Single-Token Coordinate Representation in Modality-Decoupled VLMs
This paper presents PhysMoE, a new architecture for Embodied Vision-Language Models (VLMs) designed to improve coordinate prediction in robotic grounding tasks. The authors address limitations in existing models where discrete tokenization disrupts spatial continuity and standard point regression struggles with multi-contact affordances. PhysMoE shares lower layers for multimodal context while routing upper layers to specialized experts for spatial and linguistic processing, encodes coordinates as single tokens using learned positional embeddings, and employs differentiable Signed Distance Field (SDF) supervision. The model achieves significant speedups and accuracy gains across multiple benchmarks while preserving multimodal understanding.
PhysMoE introduces a modality-decoupled Vision-Language Model architecture that achieves an 8.31x throughput speedup and a 15.7% average gain across nine coordinate-based grounding benchmarks while maintaining performance on eleven text-based embodied reasoning tasks.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
throughput speedup
8.31
View original evidence
atial priors to natively align coordinates with visual features and ease training; and (3) differentiable Signed Distance Field (SDF) supervision on pointing, accommodating affordances and irregular geometries without costly mask decoders. Compared to the base model, PhysMoE achieves a 15.7% average gain and an 8.31x throughput speedup across 9 coordinate-based grounding benchmarks (trajectory, affordance, and locali
Open source S4number of coordinate-based grounding benchmarks
9
View original evidence
atial priors to natively align coordinates with visual features and ease training; and (3) differentiable Signed Distance Field (SDF) supervision on pointing, accommodating affordances and irregular geometries without costly mask decoders. Compared to the base model, PhysMoE achieves a 15.7% average gain and an 8.31x throughput speedup across 9 coordinate-based grounding benchmarks (trajectory, affordance, and locali
Open source S4number of text-based embodied benchmarks
11
View original evidence
zation), while preserving multimodal understanding and reasoning across 11 text-based embodied benchmarks.
Open source S5
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
atial priors to natively align coordinates with visual features and ease training; and (3) differentiable Signed Distance Field (SDF) supervision on pointing, accommodating affordances and irregular geometries without costly mask decoders. Compared to the base model, PhysMoE achieves a 15.7% average gain and an 8.31x throughput speedup across 9 coordinate-based grounding benchmarks (trajectory, affordance, and locali
Open source S4
zation), while preserving multimodal understanding and reasoning across 11 text-based embodied benchmarks.
Open source S5
Source：arXiv Robotics — research abstracts · arxiv.org