17.Motor-Skill Policy Representations

17.1 What should the policy output?

Reacher must move its end-effector to a target. Which of these should the network produce?

• Joint torques $\tau \in \mathbb{R}^2$, at 1 kHz.
• Joint position targets $q^{\text{des}}$, tracked by a PD loop.
• End-effector velocity $\dot p$, converted through the Jacobian.
• The goal of a movement primitive, executed over a second.

All four are valid, and they produce radically different learning problems for identical hardware. Tang's survey organizes them into three levels.

17.2 Dynamic movement primitives

The most influential motor representation in robotics, and still the right answer for a surprising number of problems.

A DMP models a movement as a spring–damper system pulled toward a goal, perturbed by a learned forcing term. Two coupled systems.

The canonical system is a clock that decays monotonically to zero:

$\tau \dot x = -\alpha_x x, \qquad x(0) = 1.$

The transformation system is the movement itself:

$\tau^2 \ddot y = \alpha_y\left(\beta_y (g - y) - \tau \dot y\right) + f(x),$

where $g$ is the goal and $f(x)$ is the learned forcing term, represented as a normalized weighted sum of Gaussian basis functions:

$f(x) = \frac{\sum_i \psi_i(x)\, w_i}{\sum_i \psi_i(x)}\, x\,(g - y_0), \qquad \psi_i(x) = \exp\left(-h_i (x - c_i)^2\right).$

DMPs also generalize in two ways that fall out of the structure for free. Changing $g$ re-targets the movement while preserving its shape — one demonstration covers a continuum of targets. Changing $\tau$ rescales time without changing the path — one demonstration covers a range of speeds.

Fitting from a demonstration is a linear least-squares problem, not an RL problem. Given a recorded trajectory $y_{\text{demo}}(t)$, invert the transformation system for the forcing term it must have produced:

$f_{\text{target}}(t) = \tau^2 \ddot y_{\text{demo}} - \alpha_y\left(\beta_y(g - y_{\text{demo}}) - \tau \dot y_{\text{demo}}\right),$

then solve for $w$ by locally weighted regression. One demonstration, one linear solve, done.

use nalgebra::DVector;
 
pub struct Dmp {
    alpha_x: f64,
    alpha_y: f64,
    beta_y: f64,        // = alpha_y / 4 for critical damping
    centers: DVector<f64>,
    widths: DVector<f64>,
    pub weights: DVector<f64>,   // the learned parameters — all of them
}
 
pub struct DmpState { pub y: f64, pub dy: f64, pub x: f64 }
 
impl Dmp {
    /// Normalized basis activation, gated by the canonical clock.
    fn forcing(&self, x: f64, y0: f64, goal: f64) -> f64 {
        let psi = (&self.centers - DVector::repeat(self.centers.len(), x))
            .map(|d| d * d)
            .component_mul(&self.widths)
            .map(|v| (-v).exp());
        let den = psi.sum();
        if den < 1e-10 { return 0.0; }
        // The factor x forces f → 0 as the clock decays: this is what makes
        // Theorem 17.1 hold for ANY weights.
        (psi.dot(&self.weights) / den) * x * (goal - y0)
    }
 
    pub fn step(&self, s: &DmpState, y0: f64, goal: f64, tau: f64, dt: f64) -> DmpState {
        let f = self.forcing(s.x, y0, goal);
        let ddy = (self.alpha_y * (self.beta_y * (goal - s.y) - tau * s.dy) + f) / (tau * tau);
        DmpState {
            dy: s.dy + ddy * dt,
            y: s.y + (s.dy + ddy * dt) * dt,
            x: s.x - (self.alpha_x * s.x * dt) / tau,
        }
    }
 
    /// Fit weights from one demonstration by locally weighted regression.
    /// Note: a linear solve, not an RL problem.
    pub fn fit(&mut self, demo: &[f64], dt: f64, tau: f64) {
        let (y0, goal) = (demo[0], *demo.last().unwrap());
        let n = demo.len();
 
        let dy: Vec<f64> = (0..n).map(|i|
            if i == 0 { 0.0 } else { (demo[i] - demo[i - 1]) / dt }
        ).collect();
        let ddy: Vec<f64> = (0..n).map(|i|
            if i == 0 { 0.0 } else { (dy[i] - dy[i - 1]) / dt }
        ).collect();
 
        // Invert the transformation system for the forcing term it implies.
        let mut x = 1.0;
        let mut xs = Vec::with_capacity(n);
        let mut targets = Vec::with_capacity(n);
        for i in 0..n {
            targets.push(
                tau * tau * ddy[i]
                    - self.alpha_y * (self.beta_y * (goal - demo[i]) - tau * dy[i])
            );
            xs.push(x);
            x -= self.alpha_x * x * dt / tau;
        }
 
        // Weighted least squares, one basis function at a time.
        for k in 0..self.weights.len() {
            let (mut num, mut den) = (0.0, 0.0);
            for i in 0..n {
                let psi = (-self.widths[k] * (xs[i] - self.centers[k]).powi(2)).exp();
                let s = xs[i] * (goal - y0);
                num += psi * s * targets[i];
                den += psi * s * s;
            }
            self.weights[k] = if den.abs() > 1e-10 { num / den } else { 0.0 };
        }
    }
}

17.3 Central pattern generators

DMPs represent discrete point-to-point movements. Locomotion is rhythmic, and the natural representation is different: a bank of coupled oscillators, one per limb, whose relative phases define the gait.

Each oscillator has phase $\phi_i$ evolving as

$\dot\phi_i = \omega + \sum_j K_{ij} \sin\left(\phi_j - \phi_i - \Phi_{ij}\right),$

where $\omega$ is the base frequency and $\Phi_{ij}$ the desired phase offset between limbs $i$ and $j$. The coupling drives the system toward those offsets and holds it there — a phase-locked solution exists and is stable when the coupling gains $K_{ij}$ are large enough relative to frequency mismatch.

Gaits are then just phase-offset patterns: a trot sets diagonal pairs in phase and the two diagonals in antiphase; a bound pairs front and rear. Switching gait is changing $\Phi$, not retraining.

For learned locomotion, CPGs appear in two roles: as an explicit action space where the policy modulates frequency and amplitude, or — more commonly now — as a phase input to an otherwise-free policy, giving it a clock to organize its behaviour around. Chapter 18 uses the second.

17.4 Residual RL: keep the controller, learn the correction

Chapter 13 built classical controllers that work well when the model is right. Chapter 15 showed the model is never quite right. Residual RL takes both facts seriously:

$a = \underbrace{\pi_{\text{classical}}(s)}_{\text{known, validated}} + \underbrace{\pi_\theta(s)}_{\text{learned correction}}.$

The properties are exactly what a deployment engineer wants. At initialization $\pi_\theta \approx 0$, so the system starts at the classical controller's competence rather than at random flailing — no dangerous exploration phase. Bounding $\|\pi_\theta\| \le \Delta$ bounds how far the composite can deviate from validated behaviour, which converts an unbounded safety question into a tunable one. And the learned component only has to model what the classical controller missed, which is typically a far smaller function than the whole policy.

This is the pragmatic answer for many real deployments, and it is under-represented in the literature relative to how often it is the right choice.

17.5 Options: temporal abstraction

The highest action level is skills: policies that run for many steps and terminate on a condition. The formalism is the option, a triple $\langle \mathcal{I}, \pi_o, \beta \rangle$ — an initiation set, an internal policy, and a termination condition.

With options the problem becomes a semi-MDP, and the Bellman equation acquires a variable duration:

$Q(s, o) = \mathbb{E}\left[ r_{t+1} + \cdots + \gamma^{k-1} r_{t+k} + \gamma^k \max_{o'} Q(s_{t+k}, o') \right],$

where $k$ is the (random) number of steps the option ran.

The benefit is that a decision every 50 steps rather than every step shortens the effective horizon by that factor, which transforms credit assignment. The open question — which Tang's survey names as central — is what skills should the robot learn at all? Hand-specified skills work and require domain knowledge; discovering them automatically remains unsolved in general. Chapter 19 builds a hierarchical navigation system with hand-specified skills, and Chapter 21 returns to the discovery question.

17.6 Ball-in-a-cup: 2013 and now

Kober's survey closes with a case study: a robot swinging a ball on a string into a cup attached to its end-effector. It is a good benchmark — underactuated, contact-rich at the moment of capture, and genuinely hard to demonstrate perfectly.

The 2013 pipeline. Represent the movement as a DMP fitted from a kinesthetic demonstration. Optimize the ~30 DMP weights by episodic policy search — Kober's PoWER algorithm, an EM-based method exploiting the fact that the reward is a function of a low-dimensional parameter vector. Roughly 75 real rollouts suffice.

A modern pipeline. SAC from images with demonstrations seeded into the replay buffer, an action space of joint-position targets at 20 Hz, and domain randomization for transfer. Roughly $10^5$–$10^6$ simulated steps plus fine-tuning.

17.7 Chapter bridge

Part III is complete. Chapter 13 opened the robot; Chapter 14 named the four structural difficulties; Chapter 15 built the sim-to-real bridge; Chapter 16 brought in data the robot did not generate; and this chapter established that what the policy outputs is a design decision on par with how it learns.

Part IV puts all of it to work, one competency at a time, following the taxonomy Tang's survey uses to organize the field's real results. Chapter 18 is locomotion — the flagship, the one that reached commercial deployment, and the place where every technique in Part III appears together. Ferris learns to walk, and we build the reward function, the terrain curriculum, and the teacher–student transfer that make it happen.

References