Two New Egocentric Benchmarks Evaluate VLM Readiness for Robotics
Animesh Garg and @microagi introduce two benchmarks, MicroPlanBench and MicroAnnotationBench, to assess the performance of Vision-Language Models (VLMs) in robotics. Results show that even top models like GPT-6 Astra and Sol struggle with next-action prediction and temporal grounding, while specialized models like nanoannot-1 achieve better performance at a fraction of the cost.
The paper introduces two new egocentric benchmarks, MicroPlanBench and MicroAnnotationBench, to evaluate the readiness of Vision-Language Models (VLMs) for robotics tasks. It highlights the limitations of current models in both next-action prediction and temporal grounding, while demonstrating that specialized models like nanoannot-1 outperform generalist models at a lower cost.
Source: Animesh Garg — X Robotics · Read original article ↗
Article text · Original source · English
Two new egocentric benchmarks from @microagi evaluate VLM readiness for robotics:
MicroPlanBench and MicroAnnotationBench.
tl;dr: Semantic reasoning ≠ perceptual timing
A lot of work to do.
MicroPlanBench (consequential next-action planning)
When predicting the single valid next atomic action from 8s of egocentric video, SOTA models still fail ~1/3 of the time (GPT-6 Astra: 65.6%, GPT-6.1 Sol: 65.5% at 1/5th cost).
- GPT-6 Astra error breakdown: 60% wrong manipulation, 23% wrong hand assignment, 17% wrong object/destination.
- Across all flags, 42% of failures involve tracking the wrong hand.
https://www.microagi.ai/benchmarks/microplanbench
MicroAnnotationBench (continuous temporal grounding)
Knowing what happens ≠ knowing when
Continuous 60s+ dense manipulation video remains unsolved for generalist VLMs:
- GPT-6 Astra/Sol over-segment (~22 segments/clip vs. 13.8 human mean), collapsing precision (~39%) and strict Boundary F1 (<20%).
- Scaling Test-time compute fails to fix perceptual boundaries: scaling Gemini 4 Argon to high reasoning effort dropped its headline score by 4.9 points while costing 4.7x more
https://www.microagi.ai/benchmarks/microannotationbench
Currently, Small specialists match/beat frontier APIs at scale.
@microagi's self-hosted nanoannot-1 outperforms the frontier models (Gemini 4 Argon) across the board:
- 54.8 vs 49.6 overall;
- 32.9 vs 27.3 on strict ± 0.667s Boundary F1
- ~1/10th the cost (~$0.30 vs $3.14/video hr & >300x cheaper than GPT 6 Astra)
Solid work by @ErguzelYag, Ömer Adanir & Team.
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
over-segmentation segments per clip
≈22
View original evidence
- GPT-6 Astra/Sol over-segment (~22 segments/clip vs. 13.8 human mean), collapsing precision (~39%) and strict Boundary F1 (<20%).
Open source S13
- paper: not_reported
- code: not_reported
- weights: not_reported
- dataset: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
When predicting the single valid next atomic action from 8s of egocentric video, SOTA models still fail ~1/3 of the time (GPT-6 Astra: 65.6%, GPT-6.1 Sol: 65.5% at 1/5th cost).
Open source S6
- GPT-6 Astra error breakdown: 60% wrong manipulation, 23% wrong hand assignment, 17% wrong object/destination.
Open source S7
- Across all flags, 42% of failures involve tracking the wrong hand.
Open source S8
- GPT-6 Astra/Sol over-segment (~22 segments/clip vs. 13.8 human mean), collapsing precision (~39%) and strict Boundary F1 (<20%).
Open source S13
Today we are releasing two new benchmarks: MicroPlanBench (next-action prediction): GPT-6 Astra and GPT-6.1 Sol lead. Sol costs 1/5 as much. MicroAnnotationBench (labels + timing on long real-world clips): our nanoannot-1 beats Gemini 4 Argon, at 1/10 the cost.View the quoted post on X
Source：Animesh Garg · x.com