Roz: A Taste Test for World Model Latents
Roz: A Taste Test for World Model Latents
This paper presents Roz, a new evaluation framework for latent dynamics world models in robotics. The authors argue that current state-of-the-art models often fail to encode key physical properties like mass and friction, focusing instead on image similarity. Roz addresses this by defining precise encoding requirements and introducing scalable evaluation methods that can test physical properties directly. The benchmark includes 1734 individual scenarios across 10 contrastive tasks and 3 continuous tasks. Validation is performed on control tasks using the authors' own model, AnonWM. Results show that performance on Roz improves with training steps but is not necessarily correlated with reconstruction MSE. The paper advocates for adopting Roz's techniques to enable rapid iteration and fair comparison across models in the future.
The authors introduce Roz, a benchmark comprising 1734 scenarios across 10 contrastive and 3 continuous tasks designed to evaluate whether latent dynamics world models encode physical properties such as mass, deformation, friction, and bounciness rather than just image similarity.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
scenarios
1,734
View original evidence
achieve this by defining precisely what ought to be encoded in such a world model and introducing two new evaluation methods that can directly tease physical properties and easily scale to new ones. Our evaluation, which we call Roz, includes 10 contrastive tasks and 3 continuous tasks, totaling 1734 individual scenarios. We validate our eval on control tasks. On our own model, AnonWM, we find that performance rough
Open source S3contrastive tasks
10
View original evidence
achieve this by defining precisely what ought to be encoded in such a world model and introducing two new evaluation methods that can directly tease physical properties and easily scale to new ones. Our evaluation, which we call Roz, includes 10 contrastive tasks and 3 continuous tasks, totaling 1734 individual scenarios. We validate our eval on control tasks. On our own model, AnonWM, we find that performance rough
Open source S3continuous tasks
3
View original evidence
achieve this by defining precisely what ought to be encoded in such a world model and introducing two new evaluation methods that can directly tease physical properties and easily scale to new ones. Our evaluation, which we call Roz, includes 10 contrastive tasks and 3 continuous tasks, totaling 1734 individual scenarios. We validate our eval on control tasks. On our own model, AnonWM, we find that performance rough
Open source S3
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
achieve this by defining precisely what ought to be encoded in such a world model and introducing two new evaluation methods that can directly tease physical properties and easily scale to new ones. Our evaluation, which we call Roz, includes 10 contrastive tasks and 3 continuous tasks, totaling 1734 individual scenarios. We validate our eval on control tasks. On our own model, AnonWM, we find that performance rough
Open source S3
Roz: A Taste Test for World Model Latents
Open source S1
arXiv:2611.00612v1 Announce Type: new Abstract: State of the art latent dynamics world models achieve impressive performance for robot planning. But do they actually encode the physics of the world? In this paper, we show that popular world models today achieve strong results on tasks that reduce to image similarity but forget key details about objects in the scene like mass, deformation, friction, and bounciness. We
Open source S2
ly improves with training steps, while not necessarily being correlated with reconstruction MSE. We argue current model failures are a symptom of evaluation strategies (probes and planners) that are opaque, model-specific, and high-compute. Roz requires only a model that can ingest still frames or video, a way to extract latents, and a distance function between them, and it can be run in one pass. We hope future eval
Open source S4
uations will adopt Roz's evaluation techniques. More broadly, we hope Roz can be a case study for a future where benchmarks allow rapid iteration and comparison across models.
Open source S5
Source：arXiv Robotics — research abstracts · arxiv.org