Show, Don't Tell: Detecting Novel Objects by Watching Human Videos

This research paper introduces a self-supervised system called 'Show, Don't Tell' that enables robots to quickly identify and recognize new objects shown during human demonstrations. By training bespoke object detectors on an automatically created dataset supervised by the demonstration itself, the approach bypasses the need for complex language descriptions or expensive prompt engineering typically required by open-set detectors like Vision-Language Models (VLMs). The system was deployed on a real-world robot, where empirical results showed it significantly outperforms state-of-the-art detection and recognition methods for manipulated objects, leading to improved task completion.