Do Embodied VLMs Actually Use Their Scene-Graph Memory?
Do Embodied VLMs Actually Use Their Scene-Graph Memory?
This research paper introduces a diagnostic 'graph-swap test' for embodied vision-language models (VLMs) to determine if they truly utilize scene-graph memory or ignore it in favor of textual information. The study finds that while text-based memory delivery significantly improves route question accuracy, standard graph encoders with pooled or late fusion fail to convey graph information effectively. However, assigning individual input tokens to each node recovers some graph utility. The proposed test offers a low-cost method to validate whether a VLM relies on its scene-graph memory.
A diagnostic study introduces a graph-swap test to verify whether embodied vision-language models actually rely on scene-graph memory rather than ignoring it in favor of textual facts, revealing that standard fusion methods discard graph information while node-specific tokens recover partial utility.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
number of rooms in benchmark routes
2–7
View original evidence
introduce a graph-swap test for the last of these: each episode is given another episode's graph, and we count how many answers change. On an embodied question-answering benchmark with routes through two to seven rooms, we use it to compare ways of delivering scene-graph memory to a fine-tuned VLM. As text, the memory is used strongly, adding 30 points on route questions. As a graph, it is not, and we trace why: the
Open source S3
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
introduce a graph-swap test for the last of these: each episode is given another episode's graph, and we count how many answers change. On an embodied question-answering benchmark with routes through two to seven rooms, we use it to compare ways of delivering scene-graph memory to a fine-tuned VLM. As text, the memory is used strongly, adding 30 points on route questions. As a graph, it is not, and we trace why: the
Open source S3
encoder preserves the information, but standard pooled and late fusion discard it. Giving each node its own input token recovers part of it, a gain of 8.3 points on route questions that the swap test confirms comes from the graph. The test also exposes a shortcut that accuracy hides: when the same facts appear in the text, the model ignores its graph. The test thus offers a cheap check of whether a VLM uses its scen
Open source S4
Source：arXiv Robotics — research abstracts · arxiv.org