Embodied Memory Benchmark and System for Long-Term Interaction

New Intelligence Yuan Report

Imagine a

typical scenario

:

home robot just saw a book on the coffee table, went to the kitchen to complete several tasks, then received an instruction to retrieve the book, yet began searching room by room. It has already tried an unlocked drawer, after a while, it still returns to the same drawer repeatedly trying.

For humans, remembering where things are placed, knowing if they have been moved later, and remembering which methods don't work, are all basic abilities in daily life.

But for embodied agents, linking past observations, actions, and feedback together, and using them in the next task, remains a challenge. Existing benchmarks have started to focus on visual history and spatial state updates, but there is still a lack of unified action evaluations for the diverse memory requirements in long-term interactions.

When

AI

entering a continuously changing physical world, what kind of memory does it really need? And how can we determine whether it has truly remembered and can act accordingly?

Embodied memory requires connecting visual observations, actions, and environmental feedback, preserving state changes and causal relationships during the interaction process.

To answer this question, the OmniAI team at Zhejiang University proposed

EmbodiedMemory-Bench（EMem-Bench）

: through four types of tasks and 2,554 executable interaction samples, the system evaluates whether an agent can build memory from history and use it to complete subsequent tasks.

This work also proposes an external memory system

Embodied-Memorizer（EMem）

as well as learning and managing the use of memory

EMem-8B

model. Released about three weeks ago, the benchmark has already reached nearly 30,000 downloads on Huggingface.

Project homepage:

https://arxiv.org/abs/2609.28236

Code link:

https://zju-omniai.github.io/Embodied-Omni/EmbodiedMemoryBench/

Benchmark link:

https://github.com/ZJU-OmniAI/Embodied-Omni/tree/main/embodied_memory

The bottleneck of embodied tasks is memory

https://huggingface.co/datasets/lzLiang/EmbodiedMemoryBench

The bottleneck of embodied tasks is memory

In order to identify the difficulties in long-term embodied tasks, the research team first analyzed the performance of Gemini-3-Flash and Qwen3-VL-32B on EmbodiedBench's EB-ALF and EB-Hab

on 100 failed trajectories

. Among these, four categories of issues related to embodied memory collectively accounted for 91%:

ignoring fine-grained visual cues (35%)

: the target had previously appeared in the scene, but the model failed to remember its location, leading to repeated searches in subsequent steps.

missing environmental state changes (18%)

: objects had already been moved or processed, but the model continued to act based on the old state.

failure to record the state revealed by interactive feedback (31%)

: whether the grasp was successful, the environment's hidden state, these pieces of information need to be confirmed and remembered from the outcome of the action.

inability to transfer past experiences (7%)

: previous demonstrations or corrections had already provided clues, but after switching to a different object, the model still made the same mistake.

The proportions in the figure are derived from the analysis of the above 100 failed trajectories, illustrating issues in visual omission, state tracking, feedback recording, and experience transfer.

These phenomena point to the specific requirements of embodied memory: some information appears only briefly in a single frame; some memories need to be updated as the environment changes; some states can only be discovered through interaction; and some experiences need to be generalized from specific events and then transferred to new objects.

For example, 'the drawer cannot be opened' cannot necessarily be determined from a single image; it requires a trial and corresponding feedback. 'Dishes should be placed in this drawer' is not merely a simple record of an object's location, but rather a pattern established through multiple corrections. Embodied memory needs to retain the sources and connections of this information, allowing subsequent actions to have a basis.

EMem-Bench

Each sample in EMem-Bench provides a sequence of multimodal interaction history arranged in chronological order, followed by a subsequent task.

The agent needs to retrieve key evidence from history and perform actions such as navigation, opening containers, picking up and placing objects in a simulated environment until reaching the final state required by the task. The evaluation thus connects 'what happened in the past' with 'how to act now.'

The baseline sets four types of tasks around four memory requirements:

1.

Passive Observation

: Examines whether an agent can retain fine-grained visual information encountered incidentally while performing other tasks, and retrieve the location of relevant objects in subsequent tasks. The target may be among multiple objects, and the agent may not know in advance that it will be needed later. For example, a laptop was previously on a coffee table, and after some time, a new task requires picking it up; the agent needs to locate it using that previous observation.

2.

Dynamic Tracking

: Examines whether an agent can continuously update its understanding of the environment as the position or state of an object changes. The history may contain multiple records of the same object at different times, and the model needs to determine the current valid state to avoid acting based on outdated information. For example, a book was first on a bed, then placed in a box, and finally moved to a desk; the agent should locate it based on the latest position.

3.

Interaction Failure

Examine whether the agent can remember the environmental state and interaction constraints revealed by the outcomes of its actions, and adjust its subsequent choices accordingly. Information such as locked drawers and unavailable containers must be obtained through trial, and the feedback must correspond specifically to the object in question. For example, a failed attempt to open a drawer reveals that ``the upper drawer is locked,'' and when storing a key ring later, the agent needs to use this feedback to choose a usable lower drawer.

4.

Experience Generalization

: Examining whether an agent can generalize from multiple objects' corrective experiences to infer common patterns and apply them to new objects that have not appeared in the past. This requires the model to understand why past experiences are relevant and which contexts they can be transferred to. For example, past corrections often required putting knives and spoons into drawers, and when facing a newly appearing fork, the agent needs to transfer this household's storage habit.

The four types of tasks examine visual detail retention, latest state updates, interaction feedback memory, and learning reusable experience from past corrections.

The research team first selects navigable, interactive multi-room environments, then constructs memory cues around the four types of tasks to determine subsequent actions. AI2-THOR's single-room scenarios are combined into a virtual home, where the room state is preserved when the agent leaves and returns; ProcTHOR directly provides a multi-room residence.

Subsequently, the planner inserts interactions unrelated to the target between key cues, increasing the distance between the cues and the subsequent task. These interfering trajectories must be executable in reality, without touching key objects, leaking answers, or altering the state that the target task depends on. This way, the history becomes longer, and the evidence required to complete the task can still be clearly located.

The median history length for the first three types of tasks is

78–83 steps , and the experience generalization task reaches 111 steps ; between the key cues and the target tasks across all task families, the median interval remains 52–56 steps . Passive observation tasks' key scenes on average contain 11 visible objects , which also imposes requirements on detail memory.

Baseline covers

1,118 scenarios, 125 visible objects, 83 target objects, and 33 types of containers and carriers

. Automatic checks validate environments, trajectories, evidence, and feasible solutions, while manual review further checks whether instructions are clear, history is reasonable, cues are sufficient to support expected actions, and whether the final task state is judged correctly, ultimately retaining 2,554 entries.

Starting from multi-room environments, task-related memory cues, and interfering trajectories, after automatic validation and manual review, executable evaluation samples are formed.

Embodied-Memorizer

To explore how to improve these issues, the authors propose EMem, organizing embodied experiences into three complementary memories:

Scene Memory (Scene Memory)

Save the selected visual scene, description, and identified objects within it, and bind them to the scene to help distinguish between ``kitchen cup'' and ``bedroom cup''. Similar scenes will be filtered, and efforts will be made to retain newly added objects and useful visual details.

Spatial Memory

: Maintaining the position, relationships, and properties of objects through a physical map. After a book is moved from the bed to the desk, the new positional relationship replaces conflicting old relationships, while retaining other still valid properties, forming the latest known state of the world.

Event Memory

Record actions, results, and corrections over time, preserving interaction evidence such as ``this drawer can't be opened''. Multiple related corrections can be summarized into brief rules and linked to the original events that support them, for retrieval when facing new objects.

Scene memory retains visual and scene information, spatial memory maintains the latest world state, and event memory saves interaction results and summarizes experiences.

The entire process forms «

Observation—Write—Retrieval—Action—Feedback Update

The loop of ' . The model processes history in chronological order, at each step deciding what to write based on the current scene, the previous action, and its feedback. At this point, subsequent tasks have not yet been given, and memory must accumulate from the already occurred interactions.

After receiving a task, the model needs to select the memory to query. Finding a book requires the latest location, avoiding locked drawers requires past failure feedback, and storing new tableware may require multiple corrections to derive rules. Retrieval first filters based on structural conditions such as scenario, entity, and relationship, then supplements with text matching and necessary semantic matching, and passes the relevant evidence to the action phase.

New observations and feedback generated during execution continue to update memory. If an action fails or an important state changes, the system interrupts the unexecuted plan and re-queries and replans. In this way, the results of previous actions can influence the next decision, and memory is continuously updated through interaction.

With external memory, the model still needs to decide which information is worth saving, what the current task should query, and how to convert the retrieved results into action. The author further supervised fine-tuned Qwen3-VL-8B to obtain

EMem-8B

training around these three decisions, allowing the model to learn how to perform memory operations.

Experimental results

The experiment compared general open-source models, embodied models, closed-source multimodal models, and representative multimodal memory systems. The results show that the performance of various models on the four abilities is still uneven.

The strongest model has an average success rate of 64.2%

Under the full-context setting in the paper, the model receives the complete text history and current visual observation during execution.

Gemini-3-Flash achieved the highest average success rate of 64.2% in this setting; Gemini-2.5-Pro was 60.8%, the open-source model Qwen3.6-27B was 58.2%, and GPT-5.4 was 56.8%.

Part of the model results drawn from the paper's Table2. The average success rate is the equal-weighted average of the four tasks, with full-context using text history and current visual observation.

Even for the same model, there may be a significant gap between different memory capabilities: Qwen3-VL-32B achieved a success rate of

63.1%

in experience generalization, but dynamic tracking only reached

20.3%

. The specialized training of embodied models has not consistently translated into a long-term memory advantage, indicating that spatial perception and planning training still needs to be further covered in state maintenance and experience usage.

Under the same backbone, EMem brings a significant improvement

Fixing GPT-5.4-mini as the backbone, EMem increased the average success rate from

35.3% to 58.9%, an increase of 23.6 percentage points. In this comparison, the best results of other memory systems were 37.8% for TeleMem, and EMem's overall advantage reached 21.1 percentage points.

All systems use the same GPT-5.4-mini backbone. EMem achieved the highest average success rate among memory systems evaluated in the paper.

Improvement also appears on the open-source backbone: without specialized training, EMem brings an average success rate improvement on Qwen3-VL-8B, Qwen3-VL-32B, and Mistral-Small-3.1-24B respectively.

10.8, 14.1, and 16.4 percentage points

After memory operation training, EMem-8B achieves

46.0%, which is an improvement over the original Qwen3-VL-8B 20.6 percentage points Compared to freezing the backbone directly using EMem, it further improves 9.8 percentage points.

Ablation experiments also support the complementarity of the three memories: the average success rate of the complete system is 58.9%, which drops to

46.9%–47.9%

when any one memory is removed. Among them, dynamic tracking relies more on spatial memory, interaction failure and experience generalization rely more on event memory, and passive observation relies more on scene memory.

There are more historical images, why does the success rate drop instead?

More historical images, why does the success rate drop instead?

The author conducted a control experiment using GPT-5.4-mini on the

800 samples

with each task having 200 samples. Both settings retain the full text history and current image, and the extended setting adds as many historical images as possible allowed by the context budget. The results are as follows:

Original results from Table3 of the paper. Both settings use the same 800 samples and action budget; input tokens are the average per sample.

In this set of control experiments, the average success rate dropped from

36.6% to 26.2%, a decrease of 10.4 percentage points; the average input tokens per sample increased from 41,770 to 206,489, which is approximately 4.9 times; the model delay per step increased from 4.1 seconds to 39.1 seconds, which is approximately 9.5 times. This indicates that the accumulation of original history cannot replace effective memory; the key is to retrieve task-related evidence, update outdated states, and suppress irrelevant information.

In further failure analysis, the authors examined GPT-5.4-mini on EMem-Bench

400 failed samples

with 100 examples per category. Among them

61.3%

This belongs to a memory usage error: the actions taken by the model conflict with existing historical evidence. Analysis reveals that interaction failure tasks also contain a significant amount of context reasoning errors—after remembering a failure, the model still needs to understand this feedback and form new viable decisions.

Summary and Outlook

EMem-Bench evaluates long-term embodied memory through four executable tasks, and explores memory organization, updating, and usage via EMem and EMem-8B. Making past observations, actions, and feedback the basis for subsequent decisions is the core capability this work focuses on.

Current results mainly come from a simulation environment. In the future, these capabilities need to be tested in environments with longer time spans, greater complexity, and real robots, to assess their performance in the face of sensor noise, execution errors, and changes in an open world.

Team Introduction

The OmniAI team at Zhejiang University focuses on research in large models and agents, multimodal spatial reasoning, embodied intelligence, and emotional interaction. The author, Liang Lizu, is a first-year master's student at Zhejiang University. The corresponding author is Zhang Wenqi, a researcher at Zhejiang University.

References:

https://arxiv.org/abs/2609.28236

Editor: LRST

Seconds to chase ASI

⭐

Like, share, and favorite in one click

⭐

Light up the star, lock in new intelligence yuan's fast push!