Answering Without Arriving: An Evidence Audit for Embodied Question Answering
Answering Without Arriving: An Evidence Audit for Embodied Question Answering
This research presents a multi-room benchmark with 200 test episodes and 800 questions to evaluate embodied question answering. The study reveals that answer accuracy alone cannot confirm whether an agent successfully navigated and observed the environment, as plausible answers can be derived from text or room labels without actual evidence gathering. Using a closed-loop vision-language agent, the authors show that without image inputs, the model answers 45.5% of questions, while replaying the reference trajectory raises accuracy to 57.4%, indicating that evidence exists along the route but is not gathered by the agent. The paper concludes that evidence acquisition, answer coverage, and shortcut sensitivity must be established before attributing accuracy gains to memory.
The paper introduces a multi-room benchmark and evaluation ladder to audit embodied question answering systems, demonstrating that high answer accuracy does not necessarily imply successful navigation or evidence gathering.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
test episodes
200
Reported trials
View original evidence
chmark whose 200 test episodes and 800 questions are stratified by route extent, with an evaluation ladder that holds the questions fixed while varying only the evidence supplied to the answerer, from text-only probes through closed-loop agents to replay of the reference trajectory. Questions are generated from symbolic records of rooms, visible objects, and frames, so every answer can be traced to the observations t
Open source S3questions
800
View original evidence
chmark whose 200 test episodes and 800 questions are stratified by route extent, with an evaluation ladder that holds the questions fixed while varying only the evidence supplied to the answerer, from text-only probes through closed-loop agents to replay of the reference trajectory. Questions are generated from symbolic records of rooms, visible objects, and frames, so every answer can be traced to the observations t
Open source S3families
8
View original evidence
hat support it. A closed-loop vision-language agent reaches no target in 200 episodes yet answers 45.5% of questions, within 0.1 points of the same model given no images; replaying the reference trajectory raises the same model to 57.4%, so the evidence exists along the route but is not gathered. A family-level audit then shows that access to evidence is not the same as use of it. Four of eight families are largely a
Open source S4families
4
View original evidence
hat support it. A closed-loop vision-language agent reaches no target in 200 episodes yet answers 45.5% of questions, within 0.1 points of the same model given no images; replaying the reference trajectory raises the same model to 57.4%, so the evidence exists along the route but is not gathered. A family-level audit then shows that access to evidence is not the same as use of it. Four of eight families are largely a
Open source S4
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
chmark whose 200 test episodes and 800 questions are stratified by route extent, with an evaluation ladder that holds the questions fixed while varying only the evidence supplied to the answerer, from text-only probes through closed-loop agents to replay of the reference trajectory. Questions are generated from symbolic records of rooms, visible objects, and frames, so every answer can be traced to the observations t
Open source S3
hat support it. A closed-loop vision-language agent reaches no target in 200 episodes yet answers 45.5% of questions, within 0.1 points of the same model given no images; replaying the reference trajectory raises the same model to 57.4%, so the evidence exists along the route but is not gathered. A family-level audit then shows that access to evidence is not the same as use of it. Four of eight families are largely a
Open source S4
nswerable from text, one cites frames that cannot establish its own answer, and the oracle conditions reach 59.5% on Rooms-visited by counting the supplied images. Evidence acquisition, answer coverage, and shortcut sensitivity must therefore be established before an accuracy gain can be credited to memory.
Open source S5
Source：arXiv Robotics — research abstracts · arxiv.org