18.Learning Locomotion

18.1 Why locomotion won first

Chapter 1's maturity ladder put locomotion at L4–L5 while manipulation sat at L2. That gap is not about effort or talent — it follows from four structural properties, and knowing them tells you when to expect RL to work on a new problem.

The dynamics simulate well. A walking robot is rigid bodies under gravity with intermittent point contacts. Physics engines are good at exactly that. Chapter 15 explained where they are bad — sustained sliding contact, deformation, precise friction — and locomotion mostly avoids those regimes.

Rewards are dense and writable. "Track this velocity command" gives a signal at every timestep. Compare with "assemble this device", where the reward is a single bit arriving minutes later.

Failure is cheap and recoverable. A quadruped that falls gets up. A manipulator that misjudges contact can destroy the workpiece, itself, or a person. That difference decides how freely you can explore.

The task is naturally continuous and repetitive. Millions of gait cycles are one long trajectory, which is exactly the regime where on-policy methods with massive parallelism shine.

18.2 Ferris: observations and actions

Ferris has 12 actuated joints — hip abduction, hip flexion and knee per leg.

Observations (the deployable set, what the student policy sees):

That is 47 numbers, all available on real hardware at 50 Hz.

Actions are joint position targets at 50 Hz, tracked by a 1 kHz PD loop with $K_p \approx 40$, $K_d \approx 1$. This is the mid-level choice from Chapter 17, and essentially the entire field has converged on it: the PD loop absorbs high-frequency dynamics, bounds the force the policy can command, and gives well-behaved gradients. Torque-level policies exist and learn much more slowly for no demonstrated benefit.

18.3 The reward function, term by term

Here is where locomotion papers actually differ from one another, and where most of the engineering time goes.

$r = w_1 r_{\text{vel}} + w_2 r_{\text{yaw}} - w_3 c_{\text{effort}} - w_4 c_{\text{orient}} + w_5 r_{\text{air}} - w_6 c_{\text{slip}} - w_7 c_{\text{smooth}}$

Set the effort penalty high and Ferris creeps. Set the air-time reward high and he prances, wasting energy on exaggerated swings. Drop the orientation term and he pitches forward until he falls. None of those is an algorithm failure — each is the optimizer correctly serving the objective it was given.

18.4 Gait as phase

A gait is a phase relationship between limbs. Give each foot $f$ a phase $\phi_f \in [0, 2\pi)$ advancing at frequency $\omega$, with a fixed offset $\Phi_f$ defining the gait:

$\phi_f(t) = \omega t + \Phi_f \pmod{2\pi}.$

A trot sets diagonal pairs together: $\Phi = (0, \pi, \pi, 0)$. A bound pairs front and rear: $(0, 0, \pi, \pi)$. Whether a foot should be in stance or swing is then a function of its phase.

Two ways to use this. Prescribe it — feed $(\sin\phi_f, \cos\phi_f)$ as observations and reward matching the intended contact schedule, giving fast, predictable learning of a specific gait. Or let it emerge — provide only a velocity-tracking reward and let the policy discover its own footfall pattern, which yields gaits that transition naturally with speed but takes longer and needs more careful reward shaping.

Modern systems typically prescribe a phase clock as an input while leaving the contact schedule unrewarded, getting organization without over-constraining the solution.

18.5 The terrain curriculum

Ferris cannot learn stairs by starting on stairs — he falls immediately, every episode, and there is no gradient. He must start on flat ground and progress.

Formally a curriculum schedules a distribution: at training stage $k$, terrain parameters are drawn from $p_k(\xi)$, and the schedule advances only when performance justifies it. The standard mechanism is per-environment difficulty with promotion and demotion: an environment whose robot traversed most of its terrain advances a level; one whose robot fell drops a level. Difficulty tracks competence automatically, and no global schedule needs tuning.

Terrain types progress roughly as: flat → rough (Perlin noise) → gentle slopes → steps → stairs → discrete obstacles → gaps.

18.6 The full pipeline

Assembling everything from Part III:

1. Build the simulation — Ferris in rapier3d, 12 PD joints, procedural terrain, 4096 parallel environments.
2. Randomize (Chapter 15) — masses ±20%, friction 0.4–1.2, motor gains ±30%, latency 0–30 ms, plus random pushes.
3. Train a teacher with privileged access — exact terrain heights under each foot, true friction, contact states. PPO (Chapter 10), a few billion simulated steps, hours on a GPU.
4. Distill into a student (Chapter 15) — the student sees only the 47 deployable observations plus a history window, trained by DAgger-style imitation on its own state distribution (Chapter 16).
5. Deploy zero-shot.

/// One additive term of the locomotion reward, with its physical units
/// recorded so logs and ablations remain interpretable.
pub struct RewardTerm {
    pub name: &'static str,
    pub units: &'static str,
    pub weight: f64,
    pub eval: fn(&FerrisState, &Command) -> f64,
}
 
pub fn velocity_tracking(s: &FerrisState, cmd: &Command) -> f64 {
    let err = (s.base_lin_vel[0] - cmd.vx).powi(2) + (s.base_lin_vel[1] - cmd.vy).powi(2);
    (-err / 0.25).exp()          // dimensionless, in [0, 1]
}
 
pub fn foot_air_time(s: &FerrisState, _cmd: &Command) -> f64 {
    // Reward deliberate swings — the standard cure for shuffling. Credited
    // only at touchdown, so it cannot be farmed by holding a foot up forever.
    s.feet
        .iter()
        .filter(|f| f.just_landed)
        .map(|f| f.air_time - 0.5)
        .sum::<f64>()
}
 
pub fn foot_slip(s: &FerrisState, _cmd: &Command) -> f64 {
    s.feet
        .iter()
        .filter(|f| f.in_contact)
        .map(|f| f.lin_vel.norm_squared())
        .sum::<f64>()
}
 
pub fn default_terms() -> Vec<RewardTerm> {
    vec![
        RewardTerm { name: "vel_track",  units: "-",      weight:  1.0,   eval: velocity_tracking },
        RewardTerm { name: "yaw_track",  units: "-",      weight:  0.5,   eval: yaw_tracking },
        RewardTerm { name: "effort",     units: "N²m²",   weight: -2.0e-4, eval: torque_squared },
        RewardTerm { name: "orient",     units: "-",      weight: -5.0,   eval: gravity_xy_squared },
        RewardTerm { name: "air_time",   units: "s",      weight:  1.0,   eval: foot_air_time },
        RewardTerm { name: "slip",       units: "m²/s²",  weight: -0.1,   eval: foot_slip },
        RewardTerm { name: "smooth",     units: "rad²",   weight: -0.01,  eval: action_rate_squared },
    ]
}
 
/// Sum the terms, and return the per-term breakdown for the dashboard.
/// Logging terms separately is what makes reward debugging tractable.
pub fn total(terms: &[RewardTerm], s: &FerrisState, cmd: &Command) -> (f64, Vec<f64>) {
    let parts: Vec<f64> = terms.iter().map(|t| t.weight * (t.eval)(s, cmd)).collect();
    (parts.iter().sum(), parts)
}

18.7 Bipeds and flight

Bipeds are harder for reasons that are structural rather than incidental. A quadruped in a trot always has two feet down and is statically stable much of the time; a biped in walking has long single-support phases and is never statically stable during them. Its support polygon is small — the area of one foot. Falling is expensive, since humanoids are top-heavy and fragile. And angular momentum management matters: arms and torso are not decoration, they are part of the control problem.

The recipe still applies, with more careful reward design (foot placement, centre-of-mass tracking, contact scheduling) and heavier reliance on reference motions — often motion-captured human walking — as either an imitation objective or a curriculum.

Quadrotors are the other end of the spectrum: fast, underactuated in translation, but with clean, well-modelled dynamics and no contact at all. That combination made them the setting for the most striking single result in the field — a learned policy beating human champions at drone racing on a physical track. The interesting detail is that its residual dynamics model was fitted from real flight data, which is Chapter 15's system identification doing the decisive work.

18.8 Chapter bridge

Locomotion's recipe is complete: mid-level actions on a PD loop, a dense multi-term reward whose weights are unit conversions, a terrain curriculum that manufactures a learnable gradient, randomization for transfer, and teacher–student distillation to reach a deployable observation set.

What Ferris does not have is anywhere to go. He tracks a velocity command; someone else decides what the command should be.

Chapter 19 supplies that. Navigation is the perception-heavy competency — partial observability in its natural habitat, where the state is a map the robot has never seen. Rusty graduates from Chapter 4's gridworld to continuous lidar-based navigation, and we confront the architectural question the survey finds genuinely unsettled: how much of the stack should be learned at all?

References