Drawing Lines to Control Robots: Shanghai Jiao Tong University Proposes a Sparse Input Strategy Guidance Framework

RoboPrompt team submitted

Quantum Bit | Official WeChat Account QbitAI

Looking at the robot's misaligned grasp of the cup and its difficulty in accurately inserting the bread maker's plug, people might instinctively remind it: 'Move a bit to the left, and a little further forward.' These adjustments, simple for humans, may not be so easy for robots.

What is more worrying is that humans may recognize the problem, but may not necessarily be able to help it effectively. The end-to-end strategy execution process often lacks convenient entry points for ordinary people to intervene. Once a robot makes a mistake, users may easily fall into the dilemma of 'knowing what to do but not knowing how to teach it.'

RoboPrompt

The exact issue we aim to resolve is this breakpoint: allowing users to intuitively 'prompt' the robot on its next move, just as they would draw a line, mark a position, or give a directional instruction on an image.

The system does not require modifying the underlying policy structure, nor does it require retraining the policy into a specialized guided model. Instead, during inference, it converts sparse human input into action drafts and then delegates them to the existing policy for refinement execution.

△

RoboPrompt overall framework: The system supports intuitive inputs such as point selection, trajectories, and directional commands, and uses human guidance data for subsequent policy improvements.

Core idea: Transforming human intent into executable actions

RoboPrompt adopts a two-stage design. The first stage is responsible

Understanding human intent

Phase two is responsible for

Put the intent back within the scope of strategic capabilities

The advantage of this approach is that user input can be very sparse, but the final action is still calibrated by the underlying policy trained on real data, thus balancing controllability and executability.

Generate action drafts from point, line, and direction instructions

In the first phase, a lightweight VLA model Evo-1 with 0.77B parameters is used as an intent translator. Users can indicate a target point on the camera view, representing the position the end effector should reach next; they can also draw a trajectory to indicate the path they want the robot to follow; they can also input global directional cues, such as 'move a little to the left'. The model converts these inputs into continuous 6D action blocks, which serve as the action drafts for the subsequent refinement process.

To train this translator, the research team did not rely on additional manual annotations, but instead automatically generated 'post-hoc prompts' from existing trajectories: projecting future TCP positions onto images to form points and trajectory labels; accumulating and normalizing displacement over a future period of time to form direction prompts labels. Random position and scale perturbations were also added during training to enable the model to adapt to users' imprecise inputs.

△

Prompt label generation: Construct points, trajectories, and global action supervision signals using future TCP projections.

Refine actions with a low-level policy, rather than replacing the policy

Although action drafts reflect human intent, they may lack low-level details such as grasping, obstacle avoidance, and contact. RoboPrompt's second stage integrates these by leveraging the denoising process of diffusion or flow matching policies: add the action draft to a certain noise level, then let the low-level policy gradually denoise from that state to obtain actions more aligned with the task and dynamics constraints.

Among them, Progressive Biased Denoising（PBD） Provides a key control parameter: denoising level γ.

The smaller γ is, the more the action follows human input; the larger γ is, the more the action reverts to the policy prior.

For prompts with a duration longer than a single action block, RoboPrompt gradually increases γ with each replanning step: first strongly follow user intent to correct deviations, then gradually hand back control to the original policy.

△

System architecture: The first stage converts human intent into action drafts, and the second stage refines actions using the denoising process of a low-level policy.

Smoothly transition control: NA-FRS

Directly adding noise cannot guarantee linear controllability between 'human intent—policy prior'. Experiments show that many diffusion and flow matching strategies produce significant action changes within a small number of denoising steps, with subsequent changes being relatively smaller. To address this, Noise Augmented Flow Reversal Steering（NA-FRS） First, follow the strategy dynamics to reverse-propagate from the action draft to the target noise level, then mix in a controllable proportion of random noise.

Describing its effect in text is: first, retain as much information as possible from the first stage action draft, then gradually restore pure strategy denoising behavior as γ increases. This way, the trajectory drawn by the user will not be immediately overwritten by the strategy, and the strategy's own task knowledge will not be completely overwritten.

△

Denoising and Flow Reversal Mechanism: Recovering the underlying policy prior step by step while preserving action draft information.

Using TOT to filter data usable for DAgger

RoboPrompt is not only used for real-time correction, but also employs successful guidance trajectories for subsequent strategy improvements. However, 'success' does not equate to 'suitability for training': some trajectories may contain jitter, repeated gripper movements, or detours, and directly adding them to DAgger would lead to inconsistent supervision for imitation learning.

Therefore, the paper introduces

Time-weighted Optimal Transport（TOT）

as a selection criterion. It compares the similarity between the guided trajectory and the expert demonstration in the policy embedding space, while penalizing excessively long execution times.

Only trajectories that are both close to expert behavior and do not have obvious redundant actions will proceed to the next round of online training.

Experimental results: higher success rate, fewer manual interventions

The experimental platform consists of the Flexiv Rizon4 robotic arm and the Robotiq 2F-85 gripper. The research team validated RoboPrompt on three types of underlying policies: Diffusion Policy, π0.5, and FastWAM. First, in the Insert Bread multi-objective scenario, RoboPrompt can guide the originally biased multi-modal policies toward the specified target. The Prompt Alignment for all three policy types exceeded 95%, indicating that robot actions can stably follow human prompts.

In more critical online improvement experiments, RoboPrompt continuously improves the policy through multiple rounds of DAgger. Taking π0.5 as an example, in the three real-world tasks of Insert Bread, Hang Cup, and Push Ball, the average success rate increased after 2–3 rounds of iteration

15.5%

. The average number of manual interventions dropped from

2.86 times

to

1.60 times, a reduction of 44.0%. In the Insert Bread task, the average success rate increased across the three strategies of Diffusion Policy, π0.5, and FastWAM 21.3%, with the number of interventions dropping from 2.60 times to 0.47 times, a reduction of 81.9%.

△

DAgger experiment results: After multi-round RoboPrompt guided data training, success rate increases and intervention count decreases.

The paper also conducted a small-scale user study. Six volunteers with no prior experience were able to complete the bread insertion task within 5 minutes of introduction, achieving an average task progress of 86.4%, close to the 86.7% of experienced users, and significantly higher than the no-guidance baseline of 59.3%. The main difference lies in the number of interventions: new users averaged 5.17 interventions, while experienced users had 2.72 interventions, indicating that the interface is intuitive and easy to use, but prior experience still helps users predict failure earlier.

△

User study: First-time users can achieve task progress close to that of experienced users after a brief introduction.

User study: first-time users can achieve task progress close to that of experienced users after a brief introduction.

The value of RoboPrompt lies not only in enabling robots to “understand prompts,” but more importantly, in establishing a deployment loop: robots execute autonomously, users correct deviations at extremely low cost, the system collects high-quality guidance trajectories, and then improves the next round strategy through DAgger.

This real-world deployment failure transforms risk into a data source

which helps alleviate the long-standing issue of scarce real-world interaction data for robots.

From an engineering perspective, the reusability of RoboPrompt is also very important. It is not tied to a single strategy architecture and has already been validated on diffusion strategies, VLA, and WAM-like strategies; the input method is also close to ordinary user habits, avoiding the threshold of professional teleoperation equipment. For scenarios that require human-robot collaboration, such as home services, warehouse sorting, and flexible manufacturing, these lightweight, guided interfaces can significantly reduce the cost of robot deployment.

Paper: https://arxiv.org/abs/2610.10534

Project website: https://yanwen-zou.github.io/Roboprompt-Website/

Github:https://github.com/yanwen-zou/Roboprompt

Author team: Yanwen Zou, Chenyang Shi, Guoxuan Xu, Wenye Yu, Wendi Chen, Ye Pan, Cewu Lu, Chuan Wen.

This work comes from the Lu Cewu/Wenchuan team at Shanghai Jiao Tong University. The first author, Zou Yanwen, is a second-year doctoral student jointly trained by Shanghai Jiao Tong University and Shanghai Chuangzhi College.

One-click triple action

"Like","Share","Heart"

Welcome to leave your thoughts in the comment section!

—

End

—

[Academic Submission] Please send an email during business hours to:

[email protected]

with the subject line [Submission], and tell us:

Who you are, Where you're from, Submission content Attach Project/homepage link, and Contact information.

🎓

We will (try to) reply to you promptly :)

🌟 Light up the star rating 🌟

Frontier advances in technology every day