19.Learning Navigation & Mobile Manipulation

19.1 Rusty graduates

Chapter 4's Rusty lived in a 12×9 grid and always knew which cell he occupied. The real Rusty has a 2-D lidar returning 64 noisy range measurements, wheel odometry that drifts, and a building he has never seen.

The observation is now $o_t = (\text{scan}_t, \text{goal vector}_t, \text{velocity}_t)$ — about 70 numbers, none of which is the state. The state would include his true pose and the complete geometry of the building.

This is the POMDP that Chapter 4 introduced and postponed. Here it is unavoidable, and it is worth seeing what goes wrong if you ignore it: a policy treating the current scan as state cannot distinguish two identical-looking corridors, cannot remember that it already tried the left branch, and will oscillate at symmetric junctions. Those are not tuning problems; they are consequences of a state representation that is not Markov.

19.2 Belief, and how a network learns to track it

Chapter 4 gave the repair. Maintain a belief $b_t(s) = \mathbb{P}(S_t = s \mid o_{1:t}, a_{1:t-1})$, updated by Bayes' rule, and the belief is Markov: $b_{t+1}$ depends only on $b_t$, $a_t$ and $o_{t+1}$. Planning over beliefs is therefore an ordinary MDP — over a continuous, high-dimensional space, which is why exact solutions are out of reach.

The practical answer is to let the policy learn its own belief representation. A recurrent policy carries a hidden state $h_t$ updated as $h_t = \text{GRU}(h_{t-1}, o_t, a_{t-1})$, and acts as $\pi(a_t \mid h_t)$.

Two implementation details decide whether recurrent PPO works. Sequence-based batching: hidden states must be carried across a trajectory, so batches are sequences, not shuffled transitions. And burn-in: when replaying a stored sequence, run the recurrent state forward over some initial steps without computing loss, so the hidden state is warm before gradients are taken.

19.3 The architecture question

Here the survey's finding is unusually direct: neither end-to-end nor modular wins in general. That is a real result rather than a failure to decide, and the reason is that they fail differently.

The classical modular stack — SLAM, global planner, local planner, tracking controller — is inspectable, independently testable, and needs no training data. In mapped, static, structured environments it is excellent, and reaching for RL there is a mistake.

End-to-end policies handle situations nobody wrote a cost function for. They also fail without explanation, need substantial data, and cannot be debugged stage by stage. When an end-to-end policy drives into a glass door, the diagnosis is "the network did that".

The hybrid — classical mapping and global routing, learned local navigation — dominates deployed systems, and the reason is visible in the widget: hand-tuned local planners degrade fastest as environments become crowded and dynamic, because their cost functions encode assumptions that stop holding. That is precisely the layer worth learning, and precisely the layer where a mistake is cheapest.

Social navigation deserves a note. Moving among people is not obstacle avoidance with moving obstacles: humans react to the robot, so the environment is a multi-agent system where the robot's action changes others' behaviour. That makes simulation hard for exactly the reason Chapter 1 gave — humans are the unmodelable part — and it is why social navigation sits lower on the maturity ladder. Chapter 21 treats it properly under human–robot interaction.

19.4 Mobile manipulation: base plus arm

Attach Reacher to Rusty and you have a mobile manipulator with a genuinely new problem: the base and the arm can both contribute to end-effector motion, so the system is kinematically redundant.

With combined configuration $q = (q_{\text{base}}, q_{\text{arm}})$ and a task Jacobian $J(q)$ mapping to end-effector velocity, the task $\dot p = J\dot q$ has infinitely many solutions when $\dim(q) > \dim(p)$. The standard resolution is the weighted pseudoinverse:

$\dot q = W^{-1} J^\top \left(J W^{-1} J^\top\right)^{-1} \dot p + \left(I - J^\dagger J\right) \dot q_0,$

where $W$ weights joint motion — heavy for the base, light for the arm, so small corrections use the arm and large repositioning uses the base — and the second term projects a secondary objective (joint-limit avoidance, manipulability maximization, keeping clear of a person) into the null space, where it cannot disturb the task.

19.5 Long horizons and the skill question

"Fetch the mug from the kitchen" decomposes into navigate, locate, approach, grasp, lift, navigate back, place. Minutes of operation, thousands of timesteps, and a reward that arrives once at the end. Flat RL has no chance: random exploration will not stumble on that sequence in any number of episodes.

Chapter 17's options framework is the formal response. With skills as temporally extended actions, the high-level policy decides every few seconds rather than every 20 ms, and the effective horizon shortens by two orders of magnitude. Credit assignment becomes tractable because there are only a handful of decisions to assign credit among.

Three ways to compose skills, each with a real cost. Hierarchical RL learns the selector by RL — general, and unstable, since the high level's environment changes as the low level improves. Planning over skills uses a symbolic planner with learned skills as operators — reliable, and requires hand-written preconditions and effects. End-to-end with skill priors trains one policy conditioned on a skill embedding — flexible, and data-hungry.

use burn::prelude::*;
use burn::nn::gru::{Gru, GruConfig};
 
pub struct RecurrentPolicy<B: Backend> {
    encoder: Mlp<B>,        // lidar scan → compact embedding
    gru: Gru<B>,            // the learned belief tracker
    actor_head: Mlp<B>,
    critic_head: Mlp<B>,    // may see privileged state during training (Ch 15)
    /// Diagnostic head predicting true pose from the hidden state. Not used for
    /// control — it exists to answer "is this thing actually localizing?"
    probe_head: Option<Mlp<B>>,
}
 
pub struct PolicyStep<B: Backend> {
    pub action_mean: Tensor<B, 2>,
    pub action_log_std: Tensor<B, 2>,
    pub value: Tensor<B, 1>,
    pub hidden: Tensor<B, 2>,
}
 
impl<B: Backend> RecurrentPolicy<B> {
    pub fn step(
        &self,
        scan: Tensor<B, 2>,        // [batch, n_beams]
        goal_vec: Tensor<B, 2>,    // [batch, 2] — goal in the robot frame
        prev_action: Tensor<B, 2>,
        hidden: Tensor<B, 2>,
    ) -> PolicyStep<B> {
        let obs = Tensor::cat(vec![scan, goal_vec, prev_action], 1);
        let embedded = self.encoder.forward(obs);
 
        // One GRU step: h_t = GRU(h_{t−1}, o_t). The hidden state carries
        // everything the policy has decided is worth remembering.
        let hidden = self.gru.forward(embedded.unsqueeze_dim(1), Some(hidden))
            .squeeze(1);
 
        let actor_out = self.actor_head.forward(hidden.clone());
        let (mean, log_std) = actor_out.split_with_sizes(vec![2, 2], 1)
            .into_iter()
            .collect_tuple()
            .unwrap();
 
        PolicyStep {
            action_mean: mean,
            action_log_std: log_std.clamp(-5.0, 2.0),
            value: self.critic_head.forward(hidden.clone()).squeeze(1),
            hidden,
        }
    }
 
    /// Train a linear probe from the hidden state to true pose, on frozen
    /// trajectories. High accuracy ⇒ the network learned to localize.
    pub fn probe_localization(&self, hidden: Tensor<B, 2>) -> Option<Tensor<B, 2>> {
        self.probe_head.as_ref().map(|p| p.forward(hidden))
    }
}

19.6 Chapter bridge

Navigation forced partial observability into the open and answered it with recurrence rather than explicit filtering. Mobile manipulation added redundancy — a resource, resolved analytically at the instant and strategically by learning. Long horizons pushed us to skills, and left the open question of where skills come from.

Chapter 20 takes on the competency that has resisted hardest. Manipulation is where simulators are least trustworthy, where object diversity defeats generalization, and where the field's maturity ladder tops out below the real-world tiers. It is also where every technique in this book converges: grasp scoring with the value methods of Chapter 9, contact-rich insertion with the continuous control of Chapter 11 and the impedance action spaces of Chapter 13, demonstrations from Chapter 16, and the randomization of Chapter 15 pushed to its limit.

References