AI Learning Platform Helps Robots Interpret Commands

Automatically extracted; no manual editorial approval recorded.

ible feature-editing pipeline for robot motion-planning. With FLEX, humans can interact with a robot by giving it instructions in natural, conversational language. The robot can not only follow the instructions literally but also read into the words to draw larger generalizations that help to improve its motion-planning task. The team demonstrated FLEX in several simulated scenarios, including one in which a robot se

Open source S3

rves a cup of coffee. In these virtual scenarios, a person interacts with a robotic arm by typing commands in natural language. As the robot moves the coffee cup over a cluttered table, the person might type "Don't spill on the laptop!" In response, the robot will reason about why it should avoid the laptop. For this, FLEX incorporates a large language model (LLM) - a type of AI system that is trained on enormous lib

Open source S4

raries of text, including social media posts, video transcriptions, and web content, which the LLM uses to identify larger trends, patterns, and generalizations. "LLMs have seen so many texts from the internet," says study author Nathan Dennler, a postdoc in MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL). "People might have written about, 'Oh, my phone dropped in the water and now it doesn't wo

Open source S5

rk,' or 'I spilled coffee on my computer and now it doesn't work.' So LLMs pick up on the idea that liquids plus devices leads to things not working." Through the LLM, the robot reasons that it should also avoid bumping into all electronics. In its updated path, the robot adjusts to avoid both the laptop and a nearby phone, without the person having to explicitly spell this out. "The idea is that there is some higher

Open source S6

-level intention behind the words that you say," Dennler says. "People naturally get this. But you have to carefully explain that to a robot, which takes time and effort. FLEX gives you an interface of communication with a robot to say, 'When people say this, they mean that.'" The study was led by Andreea Bobu, assistant professor of aeronautics and astronautics at MIT. Co-authors are Audrey Lee SM '26 and Ilia Sucho

Open source S7

lutsky of New York University. Beyond what we say At MIT, Bobu heads up the Collaborative Learning and Autonomy Research (CLEAR) Lab, where her group develops autonomous agents that learn to work for, with, and around people. She is focused on developing assistive and household robots that can carry out physical tasks, and reason and adapt in real-time, in response to a person's natural, conversational commands. "Rob

Open source S8

ots use math to do their trajectory planning. But we can't expect a consumer to learn to do this math in order to tell a robot what to do," says Dennler, who is a member of the CLEAR Lab. "Instead of having to get people to learn how to use a robot in a specific way, the goal is to interact with the robot through natural language." There is a large effort in the robotics community to incorporate natural language into

Open source S9

robotic learning pipelines. Various strategies include using language to clarify a task's goal, redirect a robot's trajectory, and update its understanding of what is a safe path to take. These existing methods use natural language to tell the robot how to physically perform a task. But a robot could potentially learn much more from what we tell it. Bobu and her colleagues are expanding what robots can learn from na

Open source S10