Brain-Aligned Latent Actions: Imposing Premotor Cortex Geometry on a Robot's Action Vocabulary
Brain-Aligned Latent Actions: Imposing Premotor Cortex Geometry on a Robot's Action Vocabulary
This research paper introduces a method to align a robot's discrete latent action vocabulary with the representational geometry of the macaque ventral premotor cortex (PMv). By training a tokenizer alongside a straight-through similarity loss using PMv data recorded during shape drawing, the authors achieve a representational similarity of 0.52 on five unseen shape categories. This outperforms released tokenizers (0.20-0.26) and random kinematic encoders (0.42 at the 95th percentile). However, the study notes that kinematic descriptors like initial stroke direction match PMv better (0.65), and a label-shuffled control scores 0.41. The work is validated on a CALVIN pipeline reproducing Moto's published results.
The study demonstrates that aligning a robot's latent action vocabulary with the representational geometry of macaque ventral premotor cortex (PMv) improves similarity scores to 0.52 on unseen shape categories, though performance drops significantly when tested against kinematic or language targets without prior alignment.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- paper: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
arXiv:2611.11674v1 Announce Type: new Abstract: Robot policies increasingly act through a small vocabulary of discrete latent actions learned from unlabeled video, yet training leaves open which motion receives which code. We find this choice highly unstable: two independently trained runs of Moto's latent motion tokenizer assign the same robot clips to nearly unrelated codes (adjusted Rand index 0.001), and evaluati
Open source S2
ng one tokenizer in bf16 instead of fp32 reassigns 74% of its tokens. We use this freedom to impose the representational geometry of macaque ventral premotor cortex (PMv), recorded while a monkey drew shapes, on the tokenizer's code assignment, with a straight-through similarity loss trained alongside its reconstruction objective on robot video. In one training run per target, on five shape categories the loss never
Open source S3
saw, the PMv-trained tokenizer reaches a representational similarity of 0.52 with PMv, against 0.20-0.26 for released and no-prior tokenizers and 0.42 for the 95th percentile of random kinematic encoders. Kinematic, language, and random targets do not transfer (-0.17 to -0.03). Two results limit the claim: a hand-picked kinematic descriptor, the stroke's initial direction, matches PMv better (0.65), and a label-shuff
Open source S4
led PMv control scores 0.41. We specify the downstream manipulation test, on a CALVIN pipeline that reproduces Moto's published results, before running it.
Open source S5
Brain-Aligned Latent Actions: Imposing Premotor Cortex Geometry on a Robot's Action Vocabulary
Open source S1
Source：arXiv Robotics — research abstracts · arxiv.org