Verifying Test-State Identity in Paired Evaluation of Vision-Language-Action Models
Verifying Test-State Identity in Paired Evaluation of Vision-Language-Action Models
This research abstract presents a framework for verifying test-state identity in paired evaluations of vision-language-action (VLA) models.
The study identifies a methodological flaw in simulation-based VLA evaluation where vectorized reset conditions can cause episode indices to map to different initial states, proposing an integrity framework with provenance logging and automated checking to ensure paired comparisons are valid.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Environment
- SimulationOpen source S2
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
arXiv:2611.00896v1 Announce Type: new Abstract: Simulation-based VLA evaluation often treats matching task, seed, and episode identifiers as sufficient evidence that two policies faced identical test conditions. Under specific vectorized reset and autoreset conditions, different termination timing can change subsequent initial-state assignment, so the same episode index need not mean the same test state. We present a
Open source S2
Source：arXiv Robotics — research abstracts · arxiv.org