MIT's smart system helps robots understand human intent beyond literal instructions
MIT’s smart system helps robots understand what humans mean, not just what they say
Researchers at MIT have developed FLEX, an AI system that enables robots to infer human intentions by associating objects with potential risks using large language models. Tested in simulations, the system allows a robotic arm to avoid unmentioned fragile or electronic items based on partial instructions like 'Don't spill on the laptop!', generalizing concepts such as fragility or flammability across object categories. The process involves three stages: adding specific objects to an avoidance list, generalizing their properties via GPT-4o, and masking outdated objects in real-time using computer vision. Led by Assistant Professor Andreea Bobu, the study aims to help robots adapt to heterogeneous household preferences with minimal verbal instructions.
Source: Robotics — Robot learning news discovery · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Environment
- SimulationOpen source S8
Reported numbers
Reported duration
≈5 seconds
View original evidence
d also be avoided by the robot. Finally, in the Mask stage, the robot’s standard computer vision system would be used to identify objects in the robot’s surroundings. Objects that are no longer present would be removed from the robot’s feature list, allowing the robot to update its decisions in real time based on changes to the environment. The entire process takes approximately five seconds, according to the researc
Open source S7
- paper: not_reported
- code: not_reported
- weights: not_reported
- dataset: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
hers. Simulations demonstrate broader reasoning abilities The team tested FLEX in three activities serving coffee, moving a candle and carrying a bowling ball in simulated environments. The robot was able to follow instructions involving laptops, stacks of paper and mugs and extend these to include other objects in the same category (e.g. electronic devices, flammable materials and fragile items). The MIT Generative
Open source S8
d also be avoided by the robot. Finally, in the Mask stage, the robot’s standard computer vision system would be used to identify objects in the robot’s surroundings. Objects that are no longer present would be removed from the robot’s feature list, allowing the robot to update its decisions in real time based on changes to the environment. The entire process takes approximately five seconds, according to the researc
Open source S7
arm to test FLEX. In one example, the robot was asked to carry a cup of coffee over a cluttered table. The user said, “Don’t spill on the laptop!” FLEX identified the potential risk of spilling coffee on a laptop and used a large language model (LLM) to make an association between liquids and electronic devices. The robot then avoided the laptop and a nearby phone, even though the user had not mentioned the phone in
Open source S3
MIT’s smart system helps robots understand what humans mean, not just what they say
Open source S1
MIT has developed an AI-powered system that can help robots understand human intentions, such as the implied desires of a person asking a robot to carry a cup of coffee on a cluttered table. The system, called FLEX (Flexible Language Guided Representation Editing for Adaptive Robot Behavior), is being presented at the annual Conference on Robot Learning this month. The researchers used simulated tasks with a robotic
Open source S2
the instruction. This study was led by Assistant Professor of Aeronautics and Astronautics, Andreea Bobu. Co-authors include Postdoctoral Researcher Nathan Dennler of MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), as well as Master’s student Audrey Lee SM ‘26 and NYU student Ilia Sucholutsky. Large language models (LLMs) have been trained on vast amounts of internet text and can associate obje
Open source S4
cts and situations with related information. In the case of spilled coffee or water, for example, the LLMs associate these with potential damage to electronic devices. FLEX uses these associations to help robots better anticipate potential problems with objects that the user has not explicitly mentioned. The system goes through three stages Add, Generalize, and Mask In the Add stage, FLEX identifies objects and featu
Open source S5
Source：Robotics — Robot learning news discovery · msn.com