9.Deep Value-Based Methods: DQN & Descendants

9.1 From Q-table to Q-network

The move is small to write and large in consequence. Replace $Q(s,a)$ — a table — with $Q(s,a;\theta)$, a neural network, and minimize the squared TD error:

$L(\theta) = \mathbb{E}\left[\left(R_{t+1} + \gamma \max_{a'} Q(S_{t+1}, a'; \theta) - Q(S_t, A_t; \theta)\right)^2\right].$

Then run semi-gradient descent, exactly as in Chapter 8: differentiate the prediction, not the target.

Done naively, this fails reliably. It has all three triad ingredients and two additional problems that Chapter 8's tabular-adjacent analysis did not surface.

Correlated samples. Gradient methods assume roughly independent samples. Consecutive transitions from a robot are anything but: at 100 Hz, successive states differ by a centimetre. A batch of 32 consecutive transitions carries perhaps two transitions' worth of information, and the gradient is systematically biased toward whatever the robot happens to be doing right now.

A target that runs away. The target $R + \gamma \max_{a'} Q(S', a'; \theta)$ depends on $\theta$ — the very parameters being updated. Every gradient step moves the target. It is regression where the labels change each time you fit them, and the resulting dynamics oscillate or diverge.

9.2 The two pieces of surgery

Experience replay. Store transitions $(s,a,r,s')$ in a large circular buffer and train on uniformly sampled minibatches. Three benefits follow at once: samples within a batch are decorrelated, each expensive transition is reused many times (sample efficiency, which robots care about above all else), and the update distribution is smoothed across the whole recent history rather than concentrated wherever the policy currently is — directly attacking the distribution mismatch that Chapter 8 identified as the triad's fuel.

Target networks. Keep a second, frozen copy $\theta^-$ of the parameters, used only to compute targets, and sync it every $C$ steps:

$L(\theta) = \mathbb{E}\left[\left(R + \gamma \max_{a'} Q(S', a'; \theta^-) - Q(S, A; \theta)\right)^2\right].$

Between syncs the target is constant, so each interval is honest supervised regression toward a fixed objective. This is fitted Q-iteration in disguise, and it is where the stability comes from: the feedback loop that Chapter 8 diagnosed still exists, but its gain is now throttled by $C$.

Switch each off in turn and the failure signatures are distinct. Without replay the estimate rattles — correlated batches. Without a target network it oscillates and can run away — the moving target. That the two failures look different is useful diagnostic knowledge when your own training goes wrong.

9.3 The family that fixed DQN's flaws

Double DQN. Chapter 6 proved that $\max$ over noisy estimates overestimates. DQN's target has exactly that form, so DQN systematically overestimates action values. The fix is Chapter 6's double estimator, and it is nearly free because a second network already exists:

$Y^{\text{DoubleDQN}} = R + \gamma\, Q\big(S', \arg\max_{a'} Q(S', a'; \theta);\, \theta^-\big).$

The online network selects; the target network evaluates. One line of code, and it measurably improves both value accuracy and final policy quality.

Dueling networks. Split the head into a state-value stream and an advantage stream:

$Q(s,a) = V(s) + \left(A(s,a) - \frac{1}{|\mathcal{A}|}\sum_{a'} A(s,a')\right).$

The subtraction makes the decomposition identifiable — without it, $V$ and $A$ could shift by any constant. The benefit is that $V(s)$ is learned from every transition regardless of which action was taken, which matters greatly in states where the action barely matters. For a robot, that is most states: while driving down an empty corridor, the choice of small steering correction is nearly irrelevant, but the corridor's value is not.

Prioritized replay. Sample transitions in proportion to their TD error, $P(i) \propto |\delta_i|^\omega$, so surprising transitions are revisited more. This is Chapter 7's prioritized sweeping applied to a replay buffer. Because it changes the sampling distribution, the estimate becomes biased, and the bias must be corrected with importance-sampling weights $w_i \propto (1/(N \cdot P(i)))^\beta$, with $\beta$ annealed to 1 over training.

Rainbow combines Double, dueling, prioritized replay, multi-step returns, distributional values and noisy exploration nets. The ablation is the interesting part: multi-step returns and prioritized replay contribute most, and the components are largely complementary rather than redundant.

9.4 Distributional RL: learning the whole distribution

A genuinely different idea, and the most robotics-relevant of the extensions.

Instead of learning $\mathbb{E}[G_t]$, learn the distribution of returns $Z(s,a)$. It satisfies its own recursion — the distributional Bellman equation, where $\stackrel{D}{=}$ denotes equality in distribution:

$Z(s,a) \stackrel{D}{=} R + \gamma Z(S', A').$

C51 represents $Z$ as a categorical distribution over 51 fixed atoms, applies the Bellman operator (which shifts and scales the support), projects back onto the fixed atoms, and minimizes cross-entropy.

use burn::prelude::*;
use burn::optim::{GradientsParams, Optimizer};
 
pub struct Dqn<B: Backend> {
    online: QNetwork<B>,
    target: QNetwork<B>,
    gamma: f32,
    sync_every: usize,
    steps: usize,
}
 
impl<B: AutodiffBackend> Dqn<B> {
    pub fn train_step<O: Optimizer<QNetwork<B>, B>>(
        &mut self,
        batch: &Batch<B>,
        optim: &mut O,
        lr: f64,
    ) -> f32 {
        // Q(s, a) for the actions actually taken.
        let q_pred = self
            .online
            .forward(batch.states.clone())
            .gather(1, batch.actions.clone().unsqueeze_dim(1))
            .squeeze(1);
 
        // Double DQN: the ONLINE net picks the argmax, the TARGET net scores it.
        // This is Chapter 6's double estimator, reusing a network we already have.
        let next_actions = self
            .online
            .forward(batch.next_states.clone())
            .argmax(1);
 
        let next_q = self
            .target
            .forward(batch.next_states.clone())
            .gather(1, next_actions)
            .squeeze(1);
 
        // detach(): no gradient through the target — the semi-gradient of Ch 8.
        let target = batch
            .rewards
            .clone()
            .add(next_q.mul_scalar(self.gamma).mul(batch.not_done.clone()))
            .detach();
 
        // Huber loss: bounded gradients keep one outlier transition from
        // wrecking the network, which matters when rewards are unnormalized.
        let loss = huber(q_pred - target, 1.0).mean();
 
        let grads = GradientsParams::from_grads(loss.backward(), &self.online);
        self.online = optim.step(lr, self.online.clone(), grads);
 
        self.steps += 1;
        if self.steps % self.sync_every == 0 {
            self.target = self.online.clone(); // freeze a fresh copy
        }
        loss.into_scalar().elem()
    }
}

9.5 Where value-based methods belong in robotics

Honesty is due here, because DQN's fame does not match its robotics footprint.

The hard constraint is discrete actions. Computing $\max_{a'} Q(s',a')$ requires enumerating actions. A 7-DoF arm's torque vector is continuous; discretizing each joint into even 5 levels gives $5^7 = 78{,}125$ actions per step, and the max is now the bottleneck. This is why Chapter 11 exists.

Where value-based methods genuinely fit is mid- and high-level discrete decisions: which grasp to attempt from a set of candidates, which skill to invoke next, which waypoint to pursue. Tang's survey finds exactly this pattern — off-policy value methods appear in manipulation papers with discrete or discretized action spaces, while continuous control belongs to policy-gradient and actor–critic methods.

The grasp-selection case is the clearest, and Chapter 20 builds it: a network scores candidate grasps from a depth image, and the argmax over a few hundred candidates is both tractable and exactly the right computation.

9.6 Chapter bridge

We put a network in the value function and survived the deadly triad through two pieces of engineering — replay to decorrelate and broaden the update distribution, target networks to throttle the feedback loop — then watched a decade of refinements remove the remaining flaws one at a time.

But the $\max$ over actions is a wall. Robots command continuous torques and velocities, and no amount of discretization makes that comfortable.

Chapter 10 changes the object being learned. Instead of a value function from which a policy is extracted, parameterize the policy directly and ascend the gradient of expected return. The policy gradient theorem makes that possible, continuous actions become natural rather than awkward, and the algorithm that results — PPO — is the one behind more real-world robot RL successes than every method in this chapter combined.

References