Digitizing Human Operational Experience for Robotics Training

(Original Title: Large Models Accelerate Deployment in the Physical World, How Should Human Operational Experience Be Digitally Preserved?)

Robot Large models are now being opened to developers via API. On October 13, Physical Intelligence officially released the Pi API beta version. Developers can upload their own datasets, leverage platform-hosted computing power to fine-tune models, and perform inference through interface calls. Its standardized development workflow is clear: data collection, model training, real machine testing, and iterative sample supplementation based on failure cases.

*Physical Intelligence released the Pi API beta on October 13, integrating robot data upload, model fine-tuning, and cloud inference into a standardized development process. Image source: Physical Intelligence

This reshapes the development division of labor in robot applications: foundational model providers deliver general intelligence capabilities, while application teams perform specialized optimization based on their own robot platforms, operational scenarios, and business tasks. Model capabilities can be accessed via the cloud, but on-site operational experience still relies on manual collection and structured organization.

Production line components, store shelves, and workpieces of different sizes and materials will generate differentiated operational demands. For application developers, converting on-site operational requirements into model training and validation datasets is the core task after integrating large models. The industrial value of data collection workstations becomes increasingly prominent in this process.

Vision-Language-Action models (VLA) integrate visual perception, natural language, and robot actions. The development toolchain also constructs a complete workflow around data collection, model training, evaluation, and deployment. NVIDIA's Isaac GR00T development solution incorporates demonstration collection, strategy training, and real machine testing into the same workflow.

*NVIDIA Isaac GR000T integrates data collection, strategy training, simulation evaluation, and robot deployment into an end-to-end development workflow. Image source: NVIDIA

A series of technical advances indicate that the openness of model capabilities lowers the threshold for robot application development, enabling more teams to focus on vertical scenarios; at the same time, the ability to collect, clean, and reuse task data directly determines the project's implementation progress.

DPVR RoboPilot focuses on the human-robot interaction and operational data collection phase, enabling remote operation of robots through spatial positioning, motion tracking, and human-robot interaction capabilities, empowering development teams to complete task demonstrations and full-process trajectory recording.

In a joint implementation project with Unitree's dual-arm robot, this solution was validated. Operators hold dual-hand controllers, with the E4 embodied version head-mounted display system real-time calculating the controller's pose, which is then mapped to the robot's body via PC, enabling synchronized remote operation of both arms; the system simultaneously collects the six-degree-of-freedom pose and complete motion trajectory of both hands.

The head-mounted display primarily handles spatial positioning, which can be hung on a workstation neck mount or placed on a desk. Operators only need to issue action commands via the controller, without needing to wear the device continuously, making the device highly adaptable to demonstration and data collection scenarios.

Dual-arm robot tasks demand higher data quality: dual-side actions require coordinated cooperation, and the operator's debugging involves position, orientation, and timing rhythm. Development teams need to fully retain the entire operation process, review the demonstration logic, and build high-quality training samples.

The Unitree project practice proves that DPVR has completed the integration of spatial interaction capabilities with the dual-arm robot platform, providing standardized tools for action input and trajectory collection. For teams with robot bodies and model self-research capabilities, this product fills the critical gap of human-robot demonstration.

The value of real operational data needs to be viewed within a broader context of embodied intelligence technology.

World models and simulation technology significantly expand data generation channels. At the IROS world model special workshop in October this year, human demonstration videos, robot operational samples, synthetic data, and real environment feedback were incorporated into a unified research framework, focusing on the impact of multi-source data combinations on robot strategy performance.

Related research confirms that robot learning can rely on diverse sources of experience: simulation expands extreme scenarios, generative models enrich sample diversity, and real operational records provide a basis for actionable execution. The fusion and effectiveness validation of multi-type data still require continuous iteration around business goals.

This imposes stringent engineering requirements on data collection tools: collected data must seamlessly connect with downstream data processing pipelines, the demonstration process must be reproducible, and hardware must support long-term stable operation.

In response to these needs, DPVR has conducted specialized product design for the E4 embodied version: the device connects directly to the PC via USB3.0, pose data can be integrated into Python and C++ development environments, supports pure tracking mode, and can continue to output spatial coordinates even when the display module is turned off.

These designs target three scenarios: high-speed transmission, development integration, and workstation deployment. On one hand, they meet the needs of development teams to connect with existing control, recording, and debugging programs; on the other hand, they reduce the operational burden on operators for long-term repetitive demonstration.

A mature data acquisition workstation, with comprehensive costs far exceeding the hardware procurement price. Program integration requires development time, equipment commissioning involves trial-and-error costs, and unfriendly hardware forms will continuously increase human resource losses. Therefore, Dapeng makes trade-offs in product design: retaining core capabilities of positioning and data output, while simplifying non-essential display modules, to match the dedicated scenario of teaching and data collection.

Looking at long-term iteration, the model development mode determines the long-term value of data collection tools.

When the team performs scenario-based development based on a foundational model, data collection runs throughout the entire R&D cycle: collecting demonstration samples in the project's initial stage, supplementing difficult cases after real machine testing, and re-collecting data when operating conditions or equipment change. Data collection tools need to adapt to repeated iterations, becoming a routine capability in R&D.

In the long run, enterprises can accumulate their own proprietary task datasets of processes, materials, and operational standards, and perform adaptation and optimization based on general-purpose large models. Universities, laboratories, system integrators, and robot application companies can all build their own dedicated data acquisition workstations as needed. This trend opens up a broad market for tools that connect human operational experience with model development.

To serve diverse customers, hardware adaptation and ecosystem collaboration are indispensable.

Dual-arm tasks emphasize dual-side action coordination, and fine-hand operations require additional information on finger movements. In the joint testing with Yu Die Technology and Qiu Zhi Technology, the E4 embodied version handles spatial positioning and action input, data gloves capture fine hand movements, and mechanical arms with end-effectors perform physical operations. Modular hardware combinations support teams in customizing equipment according to different tasks.

Combined with the adaptation case of Unitree dual-arm robots, it is clear that RoboPilot's product positioning is clear: based on spatial interaction technology, it supports remote operation and data collection for multi-form robots through multi-hardware combination solutions. Standardized interfaces, action mapping capabilities, and project commissioning experience lay the engineering foundation for subsequent projects.

Between foundational large models, robot bodies, data collection tools, and application teams, professional division of labor continues to refine. Dapeng's positioning is precisely the key hub in this industrial chain between humans and machines: collecting human operation inputs, supporting task demonstration, preserving complete process data, and providing high-quality materials for downstream data processing and model training.

The value of RoboPilot is not limited to a single robot demonstration. Dapeng converts its years of accumulated spatial positioning and interaction technology into standardized data collection tools for professional R&D teams; through completing robot platform adaptations such as Unitree, it solidifies the foundation for solution implementation.

Large models lay the foundation for robot intelligence, yet human operational experience in real scenarios still requires continuous digital recording. Dapeng is investing its spatial interaction technology in this field: connecting the data pipeline from human operations to robot development, supporting embodied intelligence to transition from theoretical models to on-site implementation, and providing mature, usable tools and engineering support.