1.Why Reinforcement Learning for Robotics?

1.1 See, think, act

Every robot that does anything useful runs the same loop. It senses the world, it decides what to do, and it acts — changing the world, which it then senses again. Cameras, lidar, and joint encoders feed the first stage; planners, controllers, and policies occupy the second; motors and grippers execute the third.

Write it down with a little more care. At discrete time step $t$, the robot receives an observation $o_t$, chooses an action

$a_t = \pi(o_t),$

and the world advances to a new configuration influenced by $a_t$ together with whatever disturbances the universe supplies. The function $\pi$ is called a policy, and essentially all of robotics is the search for good policies.

Three words in that paragraph are doing unearned work, and it is worth naming the debt now.

The loop looks innocent because we have all watched robots work. The interesting question is what happens to it when the world stops cooperating.

1.2 Why direct programming breaks

Meet Rusty, a differential-drive mobile robot and the first member of this book's cast. Rusty's job is to reach a goal without hitting anything. His controller is the textbook potential-field method: steer toward the goal, steer away from obstacles, sum the two.

It is a good controller. It is also about to fail in front of you.

Nothing went wrong with the code. The potential-field method is correct, well understood, and provably convergent under assumptions — assumptions the concave trap violates. And here is the uncomfortable part: you fixed it instantly. You looked at the screen, saw that backing up was progress, and drove out. Encoding that judgment — for every trap, in every room, under every lighting condition — is the job direct programming asks of us.

A controller is a function $\pi : \mathcal{O} \to \mathcal{A}$ from observations to actions. Programming means specifying $\pi$ by hand, case by case. The effort required grows not with the size of the code but with the diversity of inputs the robot actually encounters. Four axes generate that diversity, and they explain why some robots shipped decades ago while others remain research projects:

Robots that shipped commercially live at the left of every row. Robots that remain research projects live at the right of at least one.

Here is what Rusty's world actually looks like in code — the unicycle kinematics and the controller trait that the sandbox above runs on.

pub struct Pose { pub x: f64, pub y: f64, pub theta: f64 }
pub struct Twist { pub v: f64, pub omega: f64 }   // forward m/s, yaw rad/s
 
/// Unicycle kinematics, advanced by one control period `dt` (zero-order hold).
pub fn step(p: &Pose, u: &Twist, dt: f64) -> Pose {
    Pose {
        x: p.x + u.v * p.theta.cos() * dt,
        y: p.y + u.v * p.theta.sin() * dt,
        theta: p.theta + u.omega * dt,
    }
}
 
pub trait Controller {
    fn act(&mut self, scan: &LidarScan) -> Twist;
}
 
/// The hand-coded baseline this chapter exists to break.
pub struct WallFollower { pub target_gap: f64, pub k_p: f64 }
 
impl Controller for WallFollower {
    fn act(&mut self, scan: &LidarScan) -> Twist {
        let gap = scan.min_range_deg(-95.0..=-85.0); // right-side beams
        Twist { v: 0.4, omega: self.k_p * (gap - self.target_gap) }
    }
}

Notice what the Controller trait does not say: nothing about how act arrives at its answer. A hand-written proportional rule satisfies it. So does a neural network trained for three days on a GPU cluster. That interface is the seam along which this entire book is built.

1.3 Three roads to a policy

If hand-writing $\pi$ does not scale, what does? There are exactly three families of answers, and they differ in what they demand from you rather than in how clever they are.

Demonstrate. Collect a set $D = \{(s, a)\}$ of situations and the actions an expert took in them, then fit a function to it. This is learning from demonstration, and it is supervised learning wearing a robotics hat. Its preconditions are strong and worth stating aloud: a teacher must exist, and demonstration must be physically possible. You can demonstrate driving. Demonstrating a backflip on a quadruped, or the precise force profile of an insertion, is another matter.

Reinforce. Provide no teacher at all — only a scalar judgment of how well things are going. The robot practices, receives reward, and improves. This is reinforcement learning. The precondition is subtler: you must be able to write down what success means as a number. Reward specifies what you want, never how to achieve it, and that gap is simultaneously the method's power and its most reliable source of disaster (Chapter 14 catalogues the disasters).

Hybridize. Use demonstrations to get a competent starting point, then let reinforcement learning refine it beyond what the demonstrator could do. Nearly every real-world success in manipulation takes this road, and Chapters 16 and 17 build it properly.

1.4 The anatomy of an RL problem

Four ingredients, stated informally here and made precise in Chapter 4.

The agent is the thing that chooses. The environment is everything else — including, importantly, the robot's own motors, since the agent controls them only through commands that may not be obeyed exactly. The reward $r_{t+1} \in \mathbb{R}$ is a scalar arriving after action $a_t$; the indexing is deliberate and follows Sutton and Barto, because the reward is a consequence of the action, not a property of the situation that preceded it. The policy $\pi$ maps what the agent perceives to what it does.

From these, one derived quantity dominates everything: the return, the accumulated reward over an episode. An agent that maximizes immediate reward is myopic; an agent that maximizes return is strategic. Making that distinction rigorous — and finite, since sums of infinitely many rewards need care — occupies Chapter 4.

Sutton and Barto identify four elements of any RL system. Each lands somewhere concrete on Rusty:

• Policy — Rusty's rule for choosing a heading given what the lidar reports.
• Reward signal — $+25$ for reaching the dock, $-1$ per step elapsed, $-10$ for hitting a shelf.
• Value function — how promising a corridor is in the long run, accounting for where it leads. Not the same as its immediate reward, and vastly more useful.
• Model — a floor map, if the robot has one. Chapters 5 and 12 show what becomes possible when it does; Chapters 6 and 7 show what to do when it does not.

1.5 What actually worked

Enthusiasm is cheap. The useful question is what reinforcement learning has genuinely achieved on physical hardware, and the honest answer is: a great deal in a few competencies, and remarkably little in others.

Tang and colleagues surveyed the field with an unusually disciplined instrument — a six-level maturity ladder running from "validated only in simulation" to "deployed in a commercial product". It is worth internalizing, because it converts vague claims into checkable ones, and because this book applies it to its own capstone in Chapter 22.

Read the pattern rather than the individual entries. Locomotion reached commercial deployment because its dynamics simulate faithfully, its rewards shape naturally, and a stumble costs a fall rather than a lawsuit. Manipulation stalls below the real-world tiers because object diversity defeats simulation and contact is where physics engines are least trustworthy. Human–robot interaction sits near the bottom for a reason no algorithm will fix: humans are the part of the environment nobody can simulate, so neither zero-shot transfer nor cheap practice is available.

That spread — not any particular algorithm — is the map this book navigates. Part IV devotes a chapter to each region of it.

1.6 The method, the cast, and the contract

Three commitments run through every remaining chapter.

Foundation. Derivations are complete. When this book states a theorem, it either proves it or states it precisely and says exactly where the proof lives. "It can be shown" is not an argument, and you should not accept it from a textbook any more than from a colleague.

Conceptual. Every hard concept gets something you can manipulate. This is not decoration. A discount factor you have dragged from 0.5 to 0.99, watching the value function's reach stretch across a warehouse, is a discount factor you understand in a way that reading $1/(1-\gamma)$ does not deliver.

Practical. Every algorithm is implemented in Rust, with seeded randomness and real tests. You will finish with working code, not pseudocode with the hard parts elided.

The cast is small and fixed. Rusty you have met. Pendle, a pendulum, arrives in Chapter 2 — simple enough that we can do every calculation exactly, which makes him the perfect vehicle for the mathematics. Reacher, a two-link arm, arrives in Chapter 3 and carries the manipulation thread. Ferris, a quadruped, arrives in Chapter 15 and gets a full chapter of his own plus the capstone.

1.7 Chapter bridge

We have made claims and issued IOUs. Let us be precise about which is which.

Claimed and defended: hand-written controllers fail through input diversity rather than programmer error; there are three roads to a policy with different preconditions; robot RL's real-world successes cluster in competencies that simulate well.

Still owed: the loop is not yet mathematics. "Reward" is not yet a random variable. "The agent eventually finds the best policy" is not yet a limit theorem — and until it is, it is marketing.

Chapter 2 pays these debts. It supplies probability spaces and conditional expectation, contraction mappings and the fixed-point theorem that every convergence proof in this book eventually invokes, the discretization that turns a robot's continuous dynamics into discrete decisions, and the Robbins–Monro theorem — the single result that makes learning from noisy samples work at all. Pendle arrives to carry all of it, because he is simple enough that we can compute everything exactly and check the theory against arithmetic.

References