Answering Without Arriving: An Evidence Audit for Embodied Question Answering
Overview
This research presents a multi-room benchmark with 200 test episodes and 800 questions to evaluate embodied question answering. The study reveals that answer accuracy alone cannot confirm whether an agent successfully navigated and observed the environment, as plausible answers can be derived from text or room labels without actual evidence gathering. Using a closed-loop vision-language agent, the authors show that without image inputs, the model answers 45.5% of questions, while replaying the reference trajectory raises accuracy to 57.4%, indicating that evidence exists along the route but is not gathered by the agent. The paper concludes that evidence acquisition, answer coverage, and shortcut sensitivity must be established before attributing accuracy gains to memory.
Generated from attributed reports · Updated 2 hours ago
Event evidence and corrections
0 attributed source owners. Ownership does not establish independent confirmation. Quantities are reported separately and are never added together.
Reported quantity · test episodes: 200 other · reported trials
“chmark whose 200 test episodes and 800 questions are stratified by route extent, with an evaluation ladder that holds the questions fixed while varying only the evidence supplied to the answerer, from text-only probes through closed-loop agents to replay of the reference trajectory. Questions are generated from symbolic records of rooms, visible objects, and frames, so every answer can be traced to the observations t”
Exact source · revision 1Source owner not reported
Reported quantity · questions: 800 other · Basis not reported
“chmark whose 200 test episodes and 800 questions are stratified by route extent, with an evaluation ladder that holds the questions fixed while varying only the evidence supplied to the answerer, from text-only probes through closed-loop agents to replay of the reference trajectory. Questions are generated from symbolic records of rooms, visible objects, and frames, so every answer can be traced to the observations t”
Exact source · revision 1Source owner not reported
Reported quantity · families: 8 other · Basis not reported · Differing source assertions
“hat support it. A closed-loop vision-language agent reaches no target in 200 episodes yet answers 45.5% of questions, within 0.1 points of the same model given no images; replaying the reference trajectory raises the same model to 57.4%, so the evidence exists along the route but is not gathered. A family-level audit then shows that access to evidence is not the same as use of it. Four of eight families are largely a”
Exact source · revision 1Source owner not reported
Reported quantity · families: 4 other · Basis not reported · Differing source assertions
“hat support it. A closed-loop vision-language agent reaches no target in 200 episodes yet answers 45.5% of questions, within 0.1 points of the same model given no images; replaying the reference trajectory raises the same model to 57.4%, so the evidence exists along the route but is not gathered. A family-level audit then shows that access to evidence is not the same as use of it. Four of eight families are largely a”
Exact source · revision 1Source owner not reported
Report timeline
Follow attributed reports and material updates.
- arXiv Robotics — research abstractsSignalAnswering Without Arriving: An Evidence Audit for Embodied Question Answering
This research presents a multi-room benchmark with 200 test episodes and 800 questions to evaluate embodied question answering. The study reveals that answer accuracy alone cannot confirm whether an agent successfully navigated and observed the environment, as plausible answers can be derived from text or room labels without actual evidence gathering. Using a closed-loop vision-language agent, the authors show that without image inputs, the model answers 45.5% of questions, while replaying the reference trajectory raises accuracy to 57.4%, indicating that evidence exists along the route but is not gathered by the agent. The paper concludes that evidence acquisition, answer coverage, and shortcut sensitivity must be established before attributing accuracy gains to memory.
Event coverage history
There is not enough continuous observation data to show a trend.
Timezone · UTC
Article dates follow your selected timezone. Briefing editions use Hong Kong time (UTC+8).