Learning from Observation (LfO) for Robots via 3D Body Pose Estimation

Automatically extracted; no manual editorial approval recorded.

umber of "examples": By having a human show representative, clean procedures (the action of grasping and moving cleanly) just a few to a dozen times, the AI understands the "objective (task goal)." Adaptation to environmental changes: Even if the position or shape of objects changes slightly, instead of a human re-teaching it every time, the system re-recognizes the current position of the objects in real-time via ca

Open source S5

ional information, including depth, becomes crucial. Therefore, methods such as using depth cameras or reconstructing 3D positions from multiple camera feeds are used. Additionally, AI can be used to estimate 3D human body poses from standard camera footage. During recording, it is important that the hands and arms are not obscured, the entire workbench is visible, and the lighting is stable. Furthermore, to determin

Open source S10

s. The AI analyzes the recorded footage and estimates the positions of human joints such as the shoulders, elbows, wrists, and hips. Google's MediaPipe Pose Landmarker can estimate 33 landmarks on the human body. When analyzing work movements of the right arm, the three points of the right shoulder, right elbow, and right wrist are particularly important. By representing each position with X, Y, and Z 3D coordinates,

Open source S12

cess is the most critical part of this technology. Humans and robots differ in arm length, joint positions, range of motion, and the number of joints. Therefore, simply copying human joint angles to a robot does not guarantee that it can perform the same task. This is where a kinematic conversion technique called kinematic retargeting is used. First, the movement of the human wrist is converted into coordinates withi

Open source S15

to convert the coordinates based on the robot's base. After that, the angles of each joint are calculated so that the robot's end-effector reaches the target position and orientation. This calculation is called inverse kinematics. For example, in a 6-axis robot, six joint angles are calculated to generate movement targets for each motor. However, the calculation must also take into account joint range of motion, spe

Open source S17

teaching. The "Learning from Observation (LfO)" introduced here is a technology that significantly changes this method. The human operator does not need to operate the robot directly. They perform the work as usual in front of a camera. For example, grasp a metal part from a blue tray with your right hand, lift it, carry it to the gray tray on the right, and place the part. This series of tasks becomes the teaching

Open source S7

the actual machine. In more advanced systems, it is also possible to train VLA models that link vision, language, and action using multiple work videos as teaching materials. This allows for the development of robots that can adjust their work methods according to the position of parts and surrounding conditions, rather than just replaying recorded movements. Conclusion through 5 slides A human demonstrates a task, a

Open source S21