Autonomous Earthmoving: Classical Control Guidance Makes Long-Horizon Trenching Reliably Trainable

Abstract

Model-free reinforcement learning struggles on long-horizon earthmoving tasks: reward arrives long after the actions that earned it, credit assignment fails, and soil responds far less predictably than the arm's own geometry. To address this, we formulate excavator trenching as a six-stage, goal-conditioned cycle (Start, Accommodate, Dig, Lift, Swing, Dump) within a high-fidelity Vortex Studio simulation featuring multi-body dynamics, hydraulic actuation, and deformable terrain.

Our approach, Action Blending Curriculum over Residual Reinforcement Learning (ABC-RRL), blends an untuned resolved-rate IK controller with a learned Soft Actor-Critic (SAC) policy into a single action. The policy’s authority share (λ) is either held fixed or annealed from 0 to 1 across training.

Evaluated across five authority conditions and 150 validation episodes per model, classical guidance removes the downside of pure RL: 3 of 3 seeds converge at λ = 0.2 compared to just 1 of 3 unguided seeds (λ = 1.0), where failed runs perform worse than no policy at all. Furthermore, annealing authority via the curriculum retains this reliability while achieving full autonomy, reaching 94.7–97.3% success at complete authority (λ → 1.0).

These results demonstrate that classical-control guidance makes long-horizon earthmoving reliably trainable and provides a robust path toward fully autonomous excavation.