Shades of Success: Evaluating Manipulation Quality from Object-Centric Constraints
Shades of Success: Evaluating Manipulation Quality from Object-Centric Constraints
This research presents SPECTRA, a framework for evaluating the quality of robotic manipulation tasks by analyzing object-centric constraints. Using a small set of oracle demonstrations in simulation, SPECTRA identifies admissible variations in relative poses and calculates a Task Quality Index (Q) to score unseen executions. The method achieves 94% accuracy in recovering constraints across seven simulated tasks and demonstrates strong correlation with human judgments. Additionally, the study shows that selecting high-scoring demonstrations for training a diffusion policy on hardware leads to higher success rates compared to random selection.
The paper introduces SPECTRA, an object-centric framework that quantifies manipulation quality using progress-aligned relative poses derived from simulation demonstrations, achieving 94% constraint recovery accuracy and high correlation with human judgments on unseen rollouts.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
Reported numbers
correlation
0.81
View original evidence
To score an unseen execution, deviations beyond the admissible variation are summarized by a Task Quality Index Q in [0,1]. Across seven simulated tasks, SPECTRA recovers constraints with 94% accuracy using 30 demonstrations per task. On rollouts from three learned policies, its scores achieve pooled Spearman correlations with human judgments of p=0.81 over all rollouts and p=0.61 among successful rollouts, exceeding
Open source S4correlation
0.61
View original evidence
To score an unseen execution, deviations beyond the admissible variation are summarized by a Task Quality Index Q in [0,1]. Across seven simulated tasks, SPECTRA recovers constraints with 94% accuracy using 30 demonstrations per task. On rollouts from three learned policies, its scores achieve pooled Spearman correlations with human judgments of p=0.81 over all rollouts and p=0.61 among successful rollouts, exceeding
Open source S4
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
arXiv:2611.01250v1 Announce Type: new Abstract: As learned manipulation policies become increasingly capable of solving practical manipulation tasks, assessing how well a task is performed becomes a fundamental challenge. While binary labels confirm task success, they cannot distinguish successful executions with different levels of geometric precision. To address this, we introduce SPECTRA, an object-centric framewo
Open source S2
rk for quantifying execution quality in manipulation of rigid objects. From a small set of oracle demonstrations generated in simulation, SPECTRA represents object interactions through progress-aligned relative poses. SPECTRA identifies which relative-pose directions are constrained and their admissible variations from demonstrations. These constraints establish a reference for evaluating robotic manipulation tasks.
Open source S3
To score an unseen execution, deviations beyond the admissible variation are summarized by a Task Quality Index Q in [0,1]. Across seven simulated tasks, SPECTRA recovers constraints with 94% accuracy using 30 demonstrations per task. On rollouts from three learned policies, its scores achieve pooled Spearman correlations with human judgments of p=0.81 over all rollouts and p=0.61 among successful rollouts, exceeding
Open source S4
all baselines in both subsets. Using the task reference learned in simulation, we use SPECTRA to select the highest-scoring demonstrations for training a diffusion policy on hardware. Exploratory results show higher success rate than training on randomly selected demonstrations.
Open source S5
Source：arXiv Robotics — research abstracts · arxiv.org