Joint-State Inference for Autonomous Excavators Using Cab-View Vision-Control Fusion
Joint-State Inference for Autonomous Excavators Using Cab-View Vision-Control Fusion
This research presents a closed-loop autonomous excavation framework that infers excavator joint states using onboard visual observations and control-related temporal information without relying on dedicated joint sensors or external cameras. The method encodes structural cues from cab-view images, models short-term motion trends from historical joystick commands, and incorporates mechanical joint-range constraints to ensure feasible estimates. Evaluated in an AGX Dynamics for Unity simulation environment across 20 autonomous excavation trials, the framework achieved a 95% task-completion rate with an average latency of 10.11 ms per step and a mean absolute error of 0.965 degrees for bucket-joint inference.
The paper proposes a sensor-light closed-loop framework for autonomous excavators that fuses cab-view vision with control history to infer joint states, achieving a 95% task-completion rate and 10.11 ms latency in AGX Dynamics for Unity simulations across 20 trials.
Source: arXiv Robotics — research abstracts · Read original article ↗
Loading article text…
What the source reports
Publisher-reported claims, with original evidence. These results have not been independently verified by RoboSignal.
- Environment
- SimulationOpen source S4
- Control
- Reported autonomousOpen source S2
Reported numbers
latency per step
10.11
View original evidence
ate, with an average joint-state-inference-and-control latency of 10.11 ms per step. The bucket-joint inference achieved a mean absolute error (MAE) of 0.965 degrees. These results demonstrate a sensor-light and computationally efficient joint-state-inference approach that can support closed-loop autonomous excavation with reduced dependence on additional sensing infrastructure.
Open source S5MAE for bucket-joint inference
0.965
View original evidence
ate, with an average joint-state-inference-and-control latency of 10.11 ms per step. The bucket-joint inference achieved a mean absolute error (MAE) of 0.965 degrees. These results demonstrate a sensor-light and computationally efficient joint-state-inference approach that can support closed-loop autonomous excavation with reduced dependence on additional sensing infrastructure.
Open source S5trials
20
Reported trials
View original evidence
ible estimates. The current cab-view image and inferred states are supplied to an Action Chunking with Transformers (ACT)-based control policy for online action generation. Experiments in an AGX Dynamics for Unity simulation environment under diverse operating conditions evaluate joint-state inference and end-to-end task execution. Across 20 autonomous excavation trials, the framework achieved a 95% task-completion r
Open source S4
- paper: not_reported
- code: not_reported
- weights: not_reported
- dataset: not_reported
Source excerpts and review record
Automatically extracted; no manual editorial approval recorded.
ible estimates. The current cab-view image and inferred states are supplied to an Action Chunking with Transformers (ACT)-based control policy for online action generation. Experiments in an AGX Dynamics for Unity simulation environment under diverse operating conditions evaluate joint-state inference and end-to-end task execution. Across 20 autonomous excavation trials, the framework achieved a 95% task-completion r
Open source S4
arXiv:2611.13690v1 Announce Type: new Abstract: Reliable state feedback of excavator front-end mechanisms is essential for autonomous excavation, yet retrofitting existing machines with dedicated joint sensors, external cameras, or visual markers can increase deployment and maintenance burdens. This paper proposes a closed-loop autonomous excavation framework for continuous inference of excavator joint states using o
Open source S2
ate, with an average joint-state-inference-and-control latency of 10.11 ms per step. The bucket-joint inference achieved a mean absolute error (MAE) of 0.965 degrees. These results demonstrate a sensor-light and computationally efficient joint-state-inference approach that can support closed-loop autonomous excavation with reduced dependence on additional sensing infrastructure.
Open source S5
nboard visual observations and control-related temporal information. The framework encodes structural cues from the current cab-view image, models short-term motion trends from historical joystick commands, and introduces a rolling state buffer for the bucket joint to preserve temporal continuity under severe occlusion. Mechanical joint-range constraints are incorporated into the learning objective to suppress infeas
Open source S3
Source：arXiv Robotics — research abstracts · arxiv.org