16.Demonstrations, Imitation & Offline RL

16.1 Where data can come from

Reinforcement learning's default assumption is that the robot generates its own data by exploring. Chapter 14 priced that assumption honestly. Demonstrations offer a different bargain: a human supplies the data, and the robot never has to discover the behaviour from scratch.

The modalities differ in what they can express and what they cost.

16.2 Behaviour cloning, and why it fails

The obvious approach: treat $D = \{(s_i, a_i)\}$ as a supervised learning dataset and fit $\pi_\phi(s) \approx a$. This is behaviour cloning, and it is genuinely the right first thing to try — it is simple, stable, and sometimes sufficient.

But it fails in a way that supervised learning does not, and the failure is structural rather than a matter of insufficient data.

DAgger breaks the loop with an idea that is obvious in retrospect: query the expert on the states the learner actually visits. Roll out the current policy, ask the expert what it would have done at each visited state, add those labels to the dataset, and retrain. Iterate.

Because the training distribution now converges to the learner's own state distribution, the bound improves to $O(T\varepsilon)$ — linear, and DAgger comes with a no-regret guarantee. The cost is that the expert must remain available and answerable during training, which is fine for a scripted or privileged expert and expensive for a human.

16.3 Inverse RL: recover the reward instead

Behaviour cloning copies actions. Inverse reinforcement learning aims higher: infer the reward function the demonstrator was optimizing, then optimize it yourself. The prize is generalization — a reward function transfers to new situations where a copied policy would flounder.

The problem is famously ill-posed: infinitely many reward functions explain any behaviour, including the degenerate $r \equiv 0$. Maximum-entropy IRL resolves the ambiguity with a principled tie-breaker: among all reward functions consistent with the demonstrations, prefer the one whose induced trajectory distribution has maximum entropy — commit to nothing beyond what the data forces.

The model is that trajectories are exponentially preferred by return:

$p(\tau) = \frac{1}{Z}\exp\big(R_\psi(\tau)\big), \qquad Z = \int \exp\big(R_\psi(\tau)\big)\, d\tau,$

and the reward parameters are fitted by maximum likelihood on the demonstrations. The gradient takes a clean and instructive form:

$\nabla_\psi \mathcal{L} = \mathbb{E}_{\tau \sim \text{demos}}\left[\nabla_\psi R_\psi(\tau)\right] - \mathbb{E}_{\tau \sim p_\psi}\left[\nabla_\psi R_\psi(\tau)\right].$

Raise the reward of what the expert did; lower the reward of what your current model predicts. The two terms balance exactly when the model's expected features match the demonstrator's.

16.4 Offline RL: learning from a fixed dataset

Now the modern framing, and the one Tang's survey identifies as most promising for real-world robot learning.

Given a fixed dataset $D$ of transitions — from demonstrations, from earlier policies, from scripted controllers, from anything — learn the best possible policy with no further environment interaction. No exploration, no resets, no risk.

The obstacle is specific and lethal. Off-policy algorithms like SAC evaluate $Q(s, a)$ at actions $a$ proposed by the current policy. If the policy proposes an action absent from the dataset, $Q$ is an extrapolation — and neural networks extrapolate confidently and wrongly. The policy then optimizes toward that erroneous high value, queries even further out of distribution, and the estimates diverge. Running SAC on a fixed dataset typically produces a policy far worse than the data it learned from.

use burn::prelude::*;
 
pub struct Iql<B: Backend> {
    q1: Critic<B>,
    q2: Critic<B>,
    q_target: (Critic<B>, Critic<B>),
    value: ValueNet<B>,
    actor: GaussianPolicy<B>,
    /// Expectile parameter. τ = 0.5 is plain regression; τ → 1 approaches a max
    /// over the actions the dataset actually contains.
    expectile: f32,
    /// Temperature for advantage weighting; higher copies the best actions harder.
    beta: f32,
    gamma: f32,
}
 
impl<B: AutodiffBackend> Iql<B> {
    /// Asymmetric loss: underestimates are penalized more than overestimates,
    /// so V converges to an upper expectile of Q over in-dataset actions.
    fn expectile_loss(&self, diff: Tensor<B, 1>) -> Tensor<B, 1> {
        let weight = diff
            .clone()
            .lower_elem(0.0)
            .float()
            .mul_scalar(-1.0)
            .add_scalar(1.0)              // 1 where diff ≥ 0, 0 where diff < 0
            .mul_scalar(self.expectile)
            .add(
                diff.clone().lower_elem(0.0).float().mul_scalar(1.0 - self.expectile)
            );
        (weight * diff.powf_scalar(2.0)).mean()
    }
 
    pub fn value_loss(&self, batch: &Batch<B>) -> Tensor<B, 1> {
        // Q at DATASET actions only — never at actions the policy proposes.
        let q1 = self.q_target.0.forward(batch.states.clone(), batch.actions.clone());
        let q2 = self.q_target.1.forward(batch.states.clone(), batch.actions.clone());
        let q = q1.min_pair(q2).detach();
        let v = self.value.forward(batch.states.clone());
        self.expectile_loss(q - v)
    }
 
    /// Advantage-weighted regression: imitate dataset actions, weighted by how
    /// much better than average they were.
    pub fn actor_loss(&self, batch: &Batch<B>) -> Tensor<B, 1> {
        let q1 = self.q_target.0.forward(batch.states.clone(), batch.actions.clone());
        let q2 = self.q_target.1.forward(batch.states.clone(), batch.actions.clone());
        let adv = (q1.min_pair(q2) - self.value.forward(batch.states.clone())).detach();
 
        // exp(β·A), clamped so one lucky transition cannot dominate the batch.
        let weight = adv.mul_scalar(self.beta).exp().clamp_max(100.0);
        let log_prob = self.actor.log_prob(batch.states.clone(), batch.actions.clone());
        -(weight * log_prob).mean()
    }
}

16.5 Offline-to-online: the deployment recipe

The pattern that Tang's survey finds behind the most capable manipulation systems is a three-stage pipeline, and it is worth stating as a recipe.

1. Collect a few hundred demonstrations by teleoperation.
2. Pretrain offline with IQL or CQL — a competent policy with zero risk, zero resets, and no supervision.
3. Fine-tune online with a small amount of real interaction, which is where performance exceeds the demonstrator's.

Stage 3 has a well-documented failure mode. Switching directly to an online off-policy algorithm often causes an immediate performance collapse: the fresh online data is off-distribution relative to the offline dataset, value estimates lurch, and the pretrained policy is destroyed within a few hundred steps.

The mitigations are all forms of easing the transition: keep the offline data in the replay buffer and sample a mixture; anneal the pessimism coefficient down rather than dropping it; and use a conservative policy-update constraint for the first phase of online training. Getting this transition right is more consequential than the choice between CQL and IQL.

16.6 Chapter bridge

We have three ways to use data that the robot did not generate by exploring: clone the actions (cheap, and quadratically fragile), recover the reward (general, and expensive), or learn a policy offline from whatever transitions exist (the modern default). And we have the pipeline that combines them with online RL.

One question remains open, and it has been quietly present since Chapter 13. Everything here learns what to do. Nothing has examined what the policy should output — joint torques, position targets, task-space velocities, or entire parameterized movements.

Chapter 17 takes that up, and it turns out to matter as much as the learning algorithm. Kober's insight — that the representation of motor skills is a first-class design decision — survives intact into the deep era, and Tang's action-space taxonomy is its modern statement. We build dynamic movement primitives, and recreate the ball-in-a-cup experiment with both the 2013 pipeline and a modern one, honestly compared.

References