15.Simulation & the Sim-to-Real Bridge

15.1 What a physics engine actually does

A simulator advances rigid-body state by integrating Newton–Euler dynamics subject to constraints. Three components decide its fidelity, and each is a place where reality departs.

The integrator. Chapter 2 showed explicit Euler manufacturing energy on Pendle. The stability condition for an explicit scheme is roughly $\Delta t \le 2/|\lambda_{\max}|$, where $\lambda_{\max}$ is the fastest eigenvalue of the linearized dynamics. Stiff systems — high-gain contacts, rigid transmissions — have enormous $\lambda_{\max}$, forcing tiny steps. Simulators therefore use semi-implicit or fully implicit schemes, which add artificial damping: stable, and systematically wrong in a direction that makes simulated robots feel more sluggish than real ones.

The constraint solver. Joints and contacts are constraints, resolved by solving a system each step. Most engines use iterative solvers with a fixed iteration budget, so constraints are satisfied approximately. Insufficient iterations produce visible artifacts: joints that stretch, stacked objects that sink.

Contact. This is the hard one. Non-penetration and Coulomb friction together form a complementarity problem: either the bodies are separated and the normal force is zero, or they touch and the force is non-negative — and the friction force lies inside a cone whose size depends on the normal force. Written as an LCP,

$0 \le \lambda_n \perp \big(J_n v^+ + b\big) \ge 0, \qquad \|\lambda_t\| \le \mu \lambda_n,$

where $\lambda_n, \lambda_t$ are normal and tangential impulses. Exact solutions are expensive, so engines approximate — linearizing the friction cone into a pyramid, softening the complementarity with regularization parameters (erp, cfm and their relatives).

15.2 Meet Ferris

Ferris is a quadruped: four legs, three actuated joints each, twelve degrees of freedom. His state includes base position and orientation, base linear and angular velocity, joint positions and velocities — 36 numbers before any terrain sensing.

Ferris is the right robot for this chapter because he is the archetype of the field's clearest success. Quadruped locomotion reached commercial deployment through exactly the pipeline built here, and the reasons are worth naming: his dynamics are dominated by rigid-body motion, his contacts are intermittent and roughly point-like (much easier than the sustained sliding contact of manipulation), and a fall is recoverable rather than catastrophic.

His low-level control is a PD controller at each joint, following position targets:

$\tau_j = K_p (q_j^{\text{des}} - q_j) - K_d\, \dot q_j.$

The policy outputs $q^{\text{des}}$ at 50 Hz; the PD loop runs at 1 kHz. That choice is itself a piece of design — Chapter 17 examines why almost everyone makes it — and it means the policy commands positions, not torques, with the PD gains absorbing a great deal of high-frequency dynamics the policy would otherwise have to learn.

15.3 Domain randomization

Here is the central idea, and it is counterintuitive at first: do not try to build one accurate simulator. Build a distribution of inaccurate ones, and train a policy that works across all of them.

Formally, let $\xi$ parameterize the simulator — masses, friction coefficients, motor gains, latencies, sensor noise — with $\xi \sim p(\xi)$. Train

$\max_\theta\; \mathbb{E}_{\xi \sim p(\xi)}\left[ J_\xi(\pi_\theta) \right].$

If the real robot's parameters $\xi^*$ lie within the support of $p$, a policy that performs well in expectation over $p$ should perform acceptably at $\xi^*$ — without ever having seen it. That is zero-shot transfer.

The trade is visible in the widget and worth stating plainly: you buy robustness by giving up peak performance. A policy trained at one friction value is superb at exactly that value and useless a little away from it. A broadly randomized policy is merely good everywhere. Since you cannot measure the real value precisely, merely-good-everywhere is what ships.

What to randomize on Ferris: link masses and inertias (±20%), friction coefficients (0.4–1.2), motor gains and strength (±30%), actuation latency (0–30 ms), sensor noise and bias, external pushes, and terrain geometry. What not to randomize: things you can measure precisely, like link lengths from CAD. Randomizing those wastes capacity on uncertainty you do not have.

15.4 System identification: narrow the distribution

Randomization is expensive in policy capacity — the wider $p(\xi)$, the more conservative the policy. So narrow it where you can, by fitting simulator parameters to real data.

Collect trajectories from the real robot, then solve

$\xi^* = \arg\min_\xi \sum_t \left\| s_t^{\text{real}} - s_t^{\text{sim}}(\xi) \right\|^2$

over the parameters you care about. The objective is non-convex, non-differentiable through the simulator, and low-dimensional — which makes it ideal for CMA-ES, an evolutionary strategy that maintains a Gaussian over parameters and adapts its covariance to the local landscape.

The practical pipeline is: identify what you can, randomize around the residual uncertainty, and use the identification's own uncertainty to set the randomization width. That is more principled than choosing ±20% because it sounds reasonable.

use rand::{Rng, SeedableRng};
use rand_chacha::ChaCha8Rng;
use rayon::prelude::*;
 
/// Everything the simulator lets us vary — one draw per episode.
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
pub struct DomainParams {
    pub mass_scale: [f64; 13],   // base + 12 links
    pub friction: f64,
    pub motor_kp: f64,
    pub motor_kd: f64,
    pub latency_steps: usize,
    pub terrain_roughness: f64,
}
 
#[derive(Clone)]
pub struct RandomizationRanges {
    pub mass_scale: (f64, f64),
    pub friction: (f64, f64),
    pub motor_kp: (f64, f64),
    pub latency_steps: (usize, usize),
    pub terrain_roughness: (f64, f64),
}
 
impl RandomizationRanges {
    pub fn sample<R: Rng>(&self, rng: &mut R) -> DomainParams {
        DomainParams {
            mass_scale: std::array::from_fn(|_| {
                rng.gen_range(self.mass_scale.0..=self.mass_scale.1)
            }),
            friction: rng.gen_range(self.friction.0..=self.friction.1),
            motor_kp: rng.gen_range(self.motor_kp.0..=self.motor_kp.1),
            motor_kd: rng.gen_range(0.5..=2.0),
            latency_steps: rng.gen_range(self.latency_steps.0..=self.latency_steps.1),
            terrain_roughness: rng.gen_range(
                self.terrain_roughness.0..=self.terrain_roughness.1
            ),
        }
    }
}
 
/// Collect rollouts across many randomized worlds in parallel.
/// Each worker derives its stream from the run seed, so results reproduce
/// exactly regardless of how many cores are available.
pub fn parallel_rollouts(
    policy: &Policy,
    ranges: &RandomizationRanges,
    n_envs: usize,
    horizon: usize,
    seed: u64,
) -> Vec<Episode> {
    (0..n_envs)
        .into_par_iter()
        .map(|i| {
            let mut rng = ChaCha8Rng::seed_from_u64(seed.wrapping_add(i as u64));
            let params = ranges.sample(&mut rng);
            let mut env = FerrisEnv::with_params(params);
            env.rollout(policy, horizon, &mut rng)
        })
        .collect()
}

15.5 Privileged information: asymmetric actors and teacher–student

Here is the observation that unlocked rough-terrain locomotion.

In simulation you know everything: exact contact forces, the true friction coefficient, terrain height beneath every foot, the robot's precise velocity. On hardware you know almost none of it. The naive response is to train only on what the robot will have, which throws away information that would make training far easier.

Asymmetric actor–critic exploits the asymmetry. The critic sees privileged state $s^{\text{priv}}$; the actor sees only deployable observations $o$. This is legitimate: the critic is a training-time device that is discarded at deployment, and giving it privileged information reduces its variance without biasing the policy gradient — the gradient's correctness depends on the actor's conditioning, not the critic's.

Teacher–student distillation goes further, and is the ANYmal recipe:

1. Train a teacher with full privileged access — terrain map, friction, contact states. This policy learns quickly because the problem is nearly fully observed.
2. Train a student that sees only real sensors (joint encoders, IMU, and a short history), by supervised imitation of the teacher's actions:

$\min_\phi\; \mathbb{E}_{\tau \sim \pi_\phi}\left[ D_{\text{KL}}\big(\pi_{\text{teacher}}(\cdot \mid s^{\text{priv}}) \,\big\|\, \pi_\phi(\cdot \mid o_{t-H:t})\big) \right].$

3. Deploy the student.

15.6 Zero-shot or few-shot? A decision procedure

Tang's survey gives an empirical answer, and it is worth stating as a procedure.

Zero-shot transfer — train entirely in simulation, deploy without adaptation — works when the dominant dynamics simulate faithfully. That means locomotion, navigation, and flight. These are the L4 results.

Few-shot adaptation — fine-tune with a small amount of real data — is needed when simulation captures the structure but not the details: contact-rich assembly, deformable objects, anything where friction and compliance decide the outcome.

Real-world learning without simulation is necessary when there is no usable simulator at all: physical human–robot interaction, where the human is the unmodelable part. These results sit at L1–L2, and Chapter 16's offline methods are the main tool.

The decision rule: ask what fraction of your task's difficulty lies in phenomena your simulator represents faithfully. Locomotion is mostly rigid-body dynamics — simulate it. Cloth manipulation is mostly deformation — do not expect zero-shot to work.

15.7 Chapter bridge

The sim-to-real bridge is built. Randomize what you cannot measure, identify what you can, use privileged information at training time and distill it away for deployment, and choose zero-shot or few-shot by asking what your simulator actually gets right.

But some tasks have no usable simulator, and some have no writable reward. When the human is part of the environment, or when the task is easier to show than to specify, a different source of information is needed.

Chapter 16 turns to demonstrations. We prove why naive imitation fails in a way supervised learning does not — errors compound quadratically in the horizon — build the correction, and then develop offline RL: learning good policies from a fixed dataset without any environment interaction at all. It is the closest thing robot learning has to the pretraining paradigm that transformed the rest of machine learning.

References