11.Off-Policy Continuous Control: DDPG, TD3 & SAC

11.1 The sample-efficiency imperative

Chapter 10's PPO is stable, simple, and wasteful. Each batch is used for a handful of epochs and discarded, because the moment $\theta$ changes the data is off-policy and the surrogate objective stops being valid.

In simulation that is a fine trade — spin up 4096 parallel environments and collect a million transitions per minute. On a physical arm, a million transitions is months of continuous operation, several gearbox replacements, and a human supervising throughout.

Off-policy methods keep every transition in a replay buffer and reuse it indefinitely. The cost is that we re-enter the deadly triad: function approximation, bootstrapping, and off-policy data, all three at once. Chapter 9's engineering — replay to broaden the update distribution, target networks to throttle the feedback loop — carries over, and this chapter adds the fixes that continuous actions specifically require.

11.2 The deterministic policy gradient

Chapter 9's obstacle was $\max_a Q(s,a)$ over a continuous action space. The trick is to stop computing the max and learn it: maintain a deterministic policy $\mu_\theta(s)$ trained to output the maximizing action.

Two features make this the right tool. The expectation is over states only — there is no integral over actions, because the policy is deterministic, so the estimator's variance does not grow with action dimension the way the stochastic policy gradient's does. And the state distribution belongs to the behaviour policy, which is precisely what licenses off-policy training from a replay buffer.

The gradient is a chain rule through the critic: differentiate $Q$ with respect to the action, then push that back through the actor. In practice the actor's loss is simply $-Q_\phi(s, \mu_\theta(s))$, and autodiff handles the rest.

DDPG puts this together with Chapter 9's machinery: replay buffer, target networks (soft-updated by Polyak averaging $\theta^- \leftarrow \eta\theta + (1-\eta)\theta^-$ rather than periodic copying), and exploration by adding noise to the deterministic action.

11.3 Overestimation, and why continuous control makes it worse

Chapter 6 proved that $\mathbb{E}[\max_a Q] \ge \max_a \mathbb{E}[Q]$: taking the max of noisy estimates overestimates. DDPG does not compute an explicit max, but its actor is trained to find the maximizing action, which is the same operation performed by gradient ascent.

Worse, the actor is optimizing against the critic's errors. Wherever the critic has an erroneously high value — a spurious bump in a region of action space with little data — the actor will find it and exploit it, because that is exactly what "maximize $Q$" instructs it to do. The critic then bootstraps from that inflated value, and the error propagates into the next target.

The result is a feedback loop that inflates values throughout the state space. It is not a small bias; measured overestimation in DDPG can be an order of magnitude.

TD3 applies three fixes, each targeting a distinct part of the loop.

Clipped double-Q. Maintain two critics and use the minimum for the target:

$y = r + \gamma \min_{i=1,2} Q_{\phi_i^-}\big(s', \tilde a'\big).$

Taking the minimum of two independent estimates is deliberately pessimistic — it induces underestimation, which is safe because underestimation does not get amplified by the actor's maximization. This is Chapter 6's double estimator, adapted: rather than decoupling selection from evaluation, it simply refuses to believe the more optimistic critic.

Delayed policy updates. Update the actor once per two critic updates. A policy chasing a critic that is itself still moving amplifies error; letting the critic settle first breaks the resonance.

Target policy smoothing. Add clipped noise to the target action, $\tilde a' = \mu_{\theta^-}(s') + \text{clip}(\epsilon, -c, c)$. This regularizes the critic by enforcing that similar actions have similar values, which flattens the narrow spurious peaks the actor would otherwise exploit.

11.4 Maximum entropy: exploration as an objective

Now a genuinely different idea, and the one that produced the most robust algorithm in this chapter.

Standard RL maximizes expected return. Maximum-entropy RL maximizes return plus the entropy of the policy:

$J(\pi) = \mathbb{E}\left[\sum_t \gamma^t \Big( R_{t+1} + \alpha\, \mathcal{H}\big(\pi(\cdot\mid S_t)\big) \Big)\right],$

where $\alpha$ is a temperature trading reward against randomness.

The Bellman equation becomes soft, with the hard max replaced by a log-sum-exp:

$Q^{\text{soft}}(s,a) = r + \gamma\, \mathbb{E}_{s'}\left[\alpha \log \int \exp\big(Q^{\text{soft}}(s',a')/\alpha\big)\, da'\right],$

and the optimal policy is the Boltzmann distribution $\pi^*(a\mid s) \propto \exp\big(Q^{\text{soft}}(s,a)/\alpha\big)$. As $\alpha \to 0$ the log-sum-exp becomes a max and standard RL is recovered.

SAC implements this with two critics (TD3's clipped double-Q), a stochastic Gaussian actor with a tanh squashing to respect action bounds, and — crucially — automatic temperature tuning. Rather than fixing $\alpha$, solve a constrained problem: maximize return subject to the policy's entropy exceeding a target $\bar{\mathcal{H}}$. The Lagrangian dual gives

$\alpha \leftarrow \alpha - \lambda \nabla_\alpha \mathbb{E}\left[-\alpha \log \pi(a\mid s) - \alpha \bar{\mathcal{H}}\right],$

which raises $\alpha$ when the policy becomes too deterministic and lowers it when too random. Setting $\bar{\mathcal{H}} = -\dim(\mathcal{A})$ works across a wide range of tasks and removes the algorithm's most sensitive hyperparameter.

The actor's gradient uses the reparameterization trick: write $a = \tanh(\mu_\theta(s) + \sigma_\theta(s) \odot \epsilon)$ with $\epsilon \sim \mathcal{N}(0,I)$, so the randomness is an input rather than something to differentiate through. Gradients then flow directly into $\mu$ and $\sigma$, giving far lower variance than the score-function estimator of Chapter 10.

use burn::prelude::*;
 
pub struct Sac<B: Backend> {
    actor: GaussianPolicy<B>,
    q1: Critic<B>,
    q2: Critic<B>,
    q1_target: Critic<B>,
    q2_target: Critic<B>,
    log_alpha: Tensor<B, 1>,
    target_entropy: f32,   // conventionally −dim(A)
    gamma: f32,
    eta: f64,              // Polyak coefficient for soft target updates
}
 
impl<B: AutodiffBackend> Sac<B> {
    fn critic_target(&self, batch: &Batch<B>) -> Tensor<B, 1> {
        // Sample the next action from the CURRENT policy (not a target actor).
        let (next_a, next_logp) = self.actor.sample(batch.next_states.clone());
 
        let q1 = self.q1_target.forward(batch.next_states.clone(), next_a.clone());
        let q2 = self.q2_target.forward(batch.next_states.clone(), next_a);
 
        // Clipped double-Q: believe the more pessimistic critic. Underestimation
        // is safe; overestimation is what the actor learns to exploit.
        let min_q = q1.min_pair(q2);
        let alpha = self.log_alpha.clone().exp();
 
        // Soft target: the entropy term enters the bootstrap itself.
        let soft = min_q - next_logp * alpha;
        (batch.rewards.clone() + soft.mul_scalar(self.gamma) * batch.not_done.clone()).detach()
    }
 
    fn actor_loss(&self, batch: &Batch<B>) -> (Tensor<B, 1>, Tensor<B, 1>) {
        // Reparameterized sample: a = tanh(μ + σ ⊙ ε), so gradients flow into
        // μ and σ directly rather than through a score-function estimator.
        let (a, logp) = self.actor.sample(batch.states.clone());
 
        let q1 = self.q1.forward(batch.states.clone(), a.clone());
        let q2 = self.q2.forward(batch.states.clone(), a);
        let min_q = q1.min_pair(q2);
 
        let alpha = self.log_alpha.clone().exp().detach();
        let loss = (logp.clone() * alpha - min_q).mean();
        (loss, logp)
    }
 
    /// Lagrangian dual: push α up when the policy gets too deterministic.
    fn temperature_loss(&self, logp: Tensor<B, 1>) -> Tensor<B, 1> {
        let target = self.target_entropy;
        -(self.log_alpha.clone().exp() * (logp.detach() + target)).mean()
    }
}

11.5 PPO or SAC? A decision procedure

The honest answer is that both are correct choices in different regimes, and the deciding factor is almost never algorithmic elegance.

Choose PPO when simulation is cheap and parallelizable, the sim-to-real gap is the dominant risk, or you value stability and easy debugging over sample count. Locomotion trained in massively parallel simulators is PPO territory, and Chapter 18 shows why: with 4096 environments, sample efficiency stops mattering and PPO's robustness to hyperparameters is worth more.

Choose SAC when samples are genuinely expensive — real-robot learning, slow simulators, contact-rich manipulation where each rollout takes seconds — or when you plan to fine-tune from offline data (Chapter 16). SAC typically reaches good policies in 5–20× fewer environment steps.

In practice many manipulation results use SAC while most locomotion results use PPO, and that split reflects exactly this trade rather than any deep truth about the competencies.

11.6 Chapter bridge

We recovered sample efficiency by returning to off-policy learning, then spent the chapter making it survivable: clipped double-Q against overestimation, delayed updates and target smoothing against resonance, and maximum entropy turning exploration from a tuned schedule into an optimized quantity.

Every method so far — value-based or policy-based, on-policy or off — learns from experience with the environment and nothing else. The environment remains a black box that is sampled, never understood.

Chapter 12 opens the box. If the robot learns a model of its dynamics, it can rehearse in imagination: generate synthetic experience, plan through predicted futures, and extract far more from each real transition. Kober's survey calls this mental rehearsal and identifies it as one of the few genuinely effective answers to the sample-cost problem. The catch — model bias compounding with the rollout horizon — is what the chapter spends most of its length managing.

References