MIT's smart system helps robots understand human intent beyond literal speech
MIT’s smart system helps robots understand what humans mean, not just what they say
Researchers at MIT have developed FLEX, an AI system that enables robots to infer human intentions and avoid potential risks not explicitly mentioned in instructions. By leveraging large language models to associate objects with their properties (e.g., liquids with electronics), the system can generalize avoidance behaviors across similar items. Tested in simulations involving a robotic arm, FLEX successfully navigated cluttered environments by identifying and avoiding fragile or hazardous objects based on partial user input, completing its reasoning loop in about five seconds.
Source: Robotics — Robot learning news discovery · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Environment
- SimulationOpen source S2
Reported numbers
Reported duration
≈5 seconds
View original evidence
d also be avoided by the robot. Finally, in the Mask stage, the robot’s standard computer vision system would be used to identify objects in the robot’s surroundings. Objects that are no longer present would be removed from the robot’s feature list, allowing the robot to update its decisions in real time based on changes to the environment. The entire process takes approximately five seconds, according to the researc
Open source S7
- paper: not_reported
- code: not_reported
- weights: not_reported
- dataset: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
MIT has developed an AI-powered system that can help robots understand human intentions, such as the implied desires of a person asking a robot to carry a cup of coffee on a cluttered table. The system, called FLEX (Flexible Language Guided Representation Editing for Adaptive Robot Behavior), is being presented at the annual Conference on Robot Learning this month. The researchers used simulated tasks with a robotic
Open source S2
d also be avoided by the robot. Finally, in the Mask stage, the robot’s standard computer vision system would be used to identify objects in the robot’s surroundings. Objects that are no longer present would be removed from the robot’s feature list, allowing the robot to update its decisions in real time based on changes to the environment. The entire process takes approximately five seconds, according to the researc
Open source S7
arm to test FLEX. In one example, the robot was asked to carry a cup of coffee over a cluttered table. The user said, “Don’t spill on the laptop!” FLEX identified the potential risk of spilling coffee on a laptop and used a large language model (LLM) to make an association between liquids and electronic devices. The robot then avoided the laptop and a nearby phone, even though the user had not mentioned the phone in
Open source S3
hers. Simulations demonstrate broader reasoning abilities The team tested FLEX in three activities serving coffee, moving a candle and carrying a bowling ball in simulated environments. The robot was able to follow instructions involving laptops, stacks of paper and mugs and extend these to include other objects in the same category (e.g. electronic devices, flammable materials and fragile items). The MIT Generative
Open source S8
MIT’s smart system helps robots understand what humans mean, not just what they say
Open source S1
the instruction. This study was led by Assistant Professor of Aeronautics and Astronautics, Andreea Bobu. Co-authors include Postdoctoral Researcher Nathan Dennler of MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), as well as Master’s student Audrey Lee SM ‘26 and NYU student Ilia Sucholutsky. Large language models (LLMs) have been trained on vast amounts of internet text and can associate obje
Open source S4
cts and situations with related information. In the case of spilled coffee or water, for example, the LLMs associate these with potential damage to electronic devices. FLEX uses these associations to help robots better anticipate potential problems with objects that the user has not explicitly mentioned. The system goes through three stages Add, Generalize, and Mask In the Add stage, FLEX identifies objects and featu
Open source S5
Source：Robotics — Robot learning news discovery · msn.com