RoboJev-27B: An Open Vision-Language Judge for Robot Success Detection
Overview
RoboJev-27B is an open-weight vision-language model that predicts whether a robot successfully completes a task based on task instructions, video frames, and auxiliary camera data. Trained on 52,598 episodes, it achieves 95.2% balanced accuracy on held-out data and generalizes to unseen labs and community-collected episodes. The model runs in 0.26 seconds per episode on an H100 GPU and is released with inference code and evaluation data.
From arXiv Robotics — research abstracts
Event evidence and corrections
0 attributed source owners. Ownership does not establish independent confirmation. Quantities are reported separately and are never added together.
Reported quantity · success predictions: 20628 other · Basis not reported
“o infer on a single H100, which is about \$0.19 per 1,000 success predictions at \$2.69 per GPU-hour (not including media processing and serving costs). RoboJev-27B is released together with its inference code, along with a transparent evaluation: 20,628 predictions organized into 29 model/protocol entries are made available to facilitate browsing and to enable reproducible benchmarking of success detection. We also”
Exact source · revision 1Source owner not reported
Reported quantity · model protocol entries: 29 other · Basis not reported
“o infer on a single H100, which is about \$0.19 per 1,000 success predictions at \$2.69 per GPU-hour (not including media processing and serving costs). RoboJev-27B is released together with its inference code, along with a transparent evaluation: 20,628 predictions organized into 29 model/protocol entries are made available to facilitate browsing and to enable reproducible benchmarking of success detection. We also”
Exact source · revision 1Source owner not reported
Reported quantity · episode records: 52598 other · Basis not reported
“h a rank-32 LoRA adapter on the language model only (keeping the vision encoder frozen) using 52,598 episode records. It also uses a source-specific KL penalty that applies only to research data, anchoring predictions closer to the original model on those sources. It uses a calibrated two-token readout that outputs success/failure probabilities in a single forward pass. On 268 held-out episodes of the same robot mode”
Exact source · revision 1Source owner not reported
Reported quantity · episodes from unseen labs/robots: 459 other · Basis not reported
“l (SO-101), RoboJev-27B attains 95.2% balanced accuracy and 0.993 AUROC. It also generalises to episodes collected by unseen labs/robots (AUROC = 0.890 over 459 episodes) as well as to clearly labelled community-collected episodes (accuracy = 93.3% over 135 episodes), where the base model attains an AUROC = 0.900 on the unseen-lab/robot subset with the same inputs. It takes 0.26 seconds per episode (median, serial) t”
Exact source · revision 1Source owner not reported
Reported quantity · community-collected episodes: 135 other · Basis not reported
“l (SO-101), RoboJev-27B attains 95.2% balanced accuracy and 0.993 AUROC. It also generalises to episodes collected by unseen labs/robots (AUROC = 0.890 over 459 episodes) as well as to clearly labelled community-collected episodes (accuracy = 93.3% over 135 episodes), where the base model attains an AUROC = 0.900 on the unseen-lab/robot subset with the same inputs. It takes 0.26 seconds per episode (median, serial) t”
Exact source · revision 1Source owner not reported
Reported quantity · inference time per episode: 0.26 other · Basis not reported
“l (SO-101), RoboJev-27B attains 95.2% balanced accuracy and 0.993 AUROC. It also generalises to episodes collected by unseen labs/robots (AUROC = 0.890 over 459 episodes) as well as to clearly labelled community-collected episodes (accuracy = 93.3% over 135 episodes), where the base model attains an AUROC = 0.900 on the unseen-lab/robot subset with the same inputs. It takes 0.26 seconds per episode (median, serial) t”
Exact source · revision 1Source owner not reported
Reported quantity · cost per 1000 success predictions: 0.19 other · Basis not reported
“o infer on a single H100, which is about \$0.19 per 1,000 success predictions at \$2.69 per GPU-hour (not including media processing and serving costs). RoboJev-27B is released together with its inference code, along with a transparent evaluation: 20,628 predictions organized into 29 model/protocol entries are made available to facilitate browsing and to enable reproducible benchmarking of success detection. We also”
Exact source · revision 1Source owner not reported
Evidence dependency changes (2)
- 2026-10-30T19:49:53.433Z · evidence updated · source revision 1. Evidence extraction was updated; current source attributions are shown above.
- 2026-10-30T19:49:36.720Z · evidence updated · source revision 1. Evidence extraction was updated; current source attributions are shown above.
Report timeline
Follow attributed reports and material updates.
- arXiv Robotics — research abstractsSignalRoboJev: An Open Vision-Language Judge for Robot Success Detection
RoboJev-27B is an open-weight vision-language model that predicts whether a robot successfully completes a task based on task instructions, video frames, and auxiliary camera data. Trained on 52,598 episodes, it achieves 95.2% balanced accuracy on held-out data and generalizes to unseen labs and community-collected episodes. The model runs in 0.26 seconds per episode on an H100 GPU and is released with inference code and evaluation data.
Timezone · UTC
Article dates follow your selected timezone. Briefing editions use Hong Kong time (UTC+8).