14.The Four Curses of Robot RL

14.1 The curse of dimensionality

Kober's first curse we have already met twice: Chapter 5 computed the wall, Chapter 8 responded to it.

A 7-DoF arm has a 14-dimensional state before you add an object. Discretize each dimension into 10 bins and you have $10^{14}$ states. The number is not merely large — it is large in a way that no hardware advance touches, because it grows exponentially in the exponent.

The correct response is not more computation but generalization: function approximation (Chapter 8) so that experience in one state informs nearby states, and structured policy representations (Chapter 17) so that the space of behaviours the robot considers is far smaller than the space of behaviours it could physically produce.

There is also a curse of dimensionality in exploration that function approximation does not address. Random torques on a 7-DoF arm produce flailing that never reaches anything interesting. The volume of state space reachable by a random walk is a vanishing fraction of the whole, and no amount of ε-greedy fixes that. This is why demonstrations (Chapter 16) and structured primitives (Chapter 17) matter so much in practice.

14.2 The curse of real-world samples

Chapter 11 motivated off-policy learning with sample cost. It is worth accounting honestly for what a sample actually costs, because the accounting changes decisions.

A single episode on a physical arm consumes wall-clock time (seconds to minutes), mechanical wear on gearboxes and cable harnesses, energy, and — most expensively — a reset. Something must return the robot and the objects to a start state. Often that something is a human, and human attention is the binding constraint on nearly every real-robot learning project.

Add safety: some transitions damage the robot, the environment, or a person. You cannot sample those freely, which means the very transitions most informative about failure are the ones you must not collect.

The responses are the architecture of this book's Part III: simulate (Chapter 15), reuse data off-policy (Chapter 11), learn from demonstrations (Chapter 16), and learn a model to extract more from each transition (Chapter 12).

14.3 The curse of under-modelling

Simulation is the standard escape from sample cost, and it introduces its own curse: the simulator is wrong.

Contact is where it is most wrong. Rigid-body contact involves complementarity conditions, friction cones, and impacts — phenomena that simulators approximate with solver parameters chosen for numerical stability rather than physical fidelity. Chapter 2 already showed a version of this in miniature: an integrator that manufactures energy.

The consequence compounds. If per-step model error is $\varepsilon$ and the dynamics have Lipschitz constant $L > 1$, then trajectory error after $H$ steps grows like $\varepsilon(L^H - 1)/(L-1)$ — the same geometric bound as Chapter 12's imagination fan, now applied to the gap between simulation and reality. A policy that exploits simulator artifacts over a long horizon will fail on hardware in exactly the way it succeeded in simulation.

14.4 The curse of goal specification

The fourth curse is the one we have avoided, and it is where most real projects actually founder.

Reward is the entire interface between what you want and what the robot optimizes. It must be a scalar, it must be computable from things the robot can sense, and it will be optimized literally — including any gap between what you wrote and what you meant.

Sparse rewards ($+1$ on success, 0 otherwise) are easy to specify honestly and nearly impossible to learn from: random exploration essentially never succeeds, so there is no gradient. Dense rewards are learnable but must be engineered, and every engineered term is a new opportunity for the optimizer to find something you did not intend.

There is one principled tool, and it is worth knowing precisely because it is the only shaping that is guaranteed safe.

14.5 Safety belongs in constraints

Practitioners routinely encode safety as a large negative reward. This is a mistake, and the reason is structural: a scalar reward forces you to price safety against task success, so there always exists a task reward large enough to justify a violation. That is precisely the wrong semantics.

The right formulation is a constrained MDP: maximize return subject to expected cost constraints,

$\max_\pi\; \mathbb{E}_\pi\!\left[\sum_t \gamma^t r_t\right] \quad \text{subject to} \quad \mathbb{E}_\pi\!\left[\sum_t \gamma^t c_i(s_t,a_t)\right] \le d_i.$

The Lagrangian $\mathcal{L}(\pi,\lambda) = J_r(\pi) - \sum_i \lambda_i (J_{c_i}(\pi) - d_i)$ turns this into a saddle-point problem: ascend on $\pi$, descend on $\lambda$. The multipliers adapt automatically — when a constraint is violated, its price rises until the policy respects it.

Notice this is the same mechanism as SAC's automatic temperature tuning in Chapter 11. Entropy there was a constraint with an adaptively priced multiplier, and safety here is the same construction with a different constrained quantity.

/// Adaptive multiplier for a single expected-cost constraint E[Σ γᵗ c] ≤ d.
pub struct LagrangianDual {
    log_lambda: f64,
    limit: f64,       // the budget d
    lr: f64,
}
 
impl LagrangianDual {
    pub fn multiplier(&self) -> f64 {
        self.log_lambda.exp()
    }
 
    /// Dual ascent: raise the price when the constraint is violated,
    /// lower it (toward zero) when there is slack.
    pub fn update(&mut self, measured_cost: f64) {
        let violation = measured_cost - self.limit;
        self.log_lambda += self.lr * violation;
        // Clamp to keep the price finite when a constraint is badly violated.
        self.log_lambda = self.log_lambda.clamp(-8.0, 4.0);
    }
 
    /// The effective reward the policy optimizes.
    pub fn shaped_reward(&self, reward: f64, cost: f64) -> f64 {
        reward - self.multiplier() * cost
    }
}

14.6 Evaluating claims honestly

The last section of this chapter is methodological, and it is the one most often skipped.

Robot RL results are hard to evaluate because runs are expensive, so sample sizes are small; seeds matter enormously, so single runs mislead; and success is often reported without confidence intervals or without stating how many attempts were made.

The standards this book holds itself to, and recommends:

Report the level of real-world success. Chapter 1's L0–L5 rubric is a checkable claim, unlike "we demonstrate real-world capability".

Report seeds. Five minimum, with the spread shown, not just the mean. Deep RL's seed variance is large enough that a three-seed comparison frequently reverses on the fourth.

Report the classical baseline. Chapter 13 built the suite for this reason.

Report the sample budget in real units. Not "500k steps" but "14 hours of robot time and 62 human-supervised resets".

Report failures. A 90% success rate means one attempt in ten failed, and how it failed is more informative than the nine that worked.

14.7 Chapter bridge

Four curses, each with a response. Dimensionality is answered by generalization and structure. Sample cost is answered by simulation, off-policy reuse, demonstrations, and models. Under-modelling is answered by randomization and system identification. Goal specification is answered by potential-based shaping where possible, by demonstrations where reward cannot be written, and by constraints where safety is at stake.

Chapter 15 takes the second and third curses head-on. If simulation is the escape from sample cost, and simulation is wrong, then the central question of practical robot learning is: how do you train in a simulator you know to be wrong, and get a policy that works anyway? The answers — domain randomization, system identification, teacher–student distillation — are the machinery behind essentially every L4 result in Chapter 1's ladder. Ferris arrives to carry them.

References