21.Frontiers: HRI, Multi-Robot & Foundation Models

21.1 Humans are the unmodelable part

Every technique in Part III depended on simulation. Domain randomization needs a simulator to randomize; teacher–student needs privileged simulator state; massively parallel PPO needs thousands of simulated worlds.

None of that is available when a human is in the loop. There is no physics engine for a person deciding to reach for the same object you were reaching for. You can model human motion kinematically, but you cannot simulate the reaction to the robot's behaviour — and that reaction is precisely what physical human–robot interaction consists of.

The consequences follow directly. Zero-shot sim-to-real is unavailable, so real-world learning is mandatory. Real-world learning with humans is slow, supervised, and safety-critical, so data is scarce. Scarce data means Chapter 16's offline methods rather than Chapter 10's on-policy ones. And the safety case means Chapter 14's constrained formulation rather than a scalar penalty.

That chain of reasoning — not any algorithmic gap — is why HRI sits at L1–L2 on Chapter 1's ladder.

21.2 Shared autonomy as goal inference

Shared autonomy has the cleanest formulation, so it is worth doing properly.

The human has a goal $g$ the robot cannot observe. The robot receives human input $u_t$ — joystick, gaze, muscle signal — which is noisy evidence about $g$. Maintain a belief over goals and update it with each input:

$b_{t+1}(g) \propto P(u_t \mid g,\, s_t)\; b_t(g).$

Then act to maximize expected value under that belief:

$a_t = \arg\max_a \sum_g b_t(g)\, Q_g(s_t, a).$

This is Chapter 4's belief-MDP machinery with the hidden variable being the human's intention rather than the robot's location. Everything transfers, including the honest observation that the belief space is continuous and exact solutions are out of reach.

21.3 Multi-robot systems

Two robots in a warehouse are not two independent problems. Each is part of the other's environment, and since both are learning, each faces a non-stationary environment — the thing every convergence proof in Part I assumed away.

The formalism is the decentralized POMDP: a shared state and transition function, but each agent $i$ has its own observation function $O_i$ and must act on its own observation alone. The team shares one reward.

That is worse than the PSPACE-hardness of single-agent POMDPs, and it is not an artifact of a bad formulation. The difficulty is intrinsic: agents must reason about what other agents know, which requires reasoning about what they observe, recursively.

Centralized training with decentralized execution is the practical response, and it is exactly Chapter 15's asymmetric actor–critic applied to a team. During training the critic sees the joint state and all agents' observations, which makes its estimates stationary and low-variance. At execution each agent runs a policy conditioned only on its own observation. MAPPO is PPO with this structure and is a strong baseline despite its simplicity.

The residual difficulty is credit assignment: the team reward does not say who contributed. The standard fix is a counterfactual baseline — compare the achieved return against what would have happened had agent $i$ acted differently, holding others fixed — which is Chapter 10's baseline argument, extended to teams.

21.4 Foundation models

The fastest-moving frontier, and the one most likely to date this chapter.

Large vision-language models bring something robot RL has always lacked: priors about the world that nobody had to encode. Four ways they are being used, in increasing order of ambition.

Reward specification. Ask a VLM whether an image shows the task completed. This attacks Chapter 14's fourth curse directly — instead of engineering a reward, describe the goal in language. It works for visually-checkable goals and fails where success is a matter of forces or internal state.

Goal and plan generation. An LLM decomposes "clear the table" into a sequence of skills the robot already has. This is Chapter 19's skill-composition problem with the composer supplied by a language model rather than learned, and it partially answers the open question of where high-level structure comes from.

Vision-language-action models. Train a single large policy on diverse robot data with language conditioning, so it maps instruction plus image to action. The bet is that scale and diversity produce generalization to novel objects and phrasings — a bet that has paid off in other domains.

RL fine-tuning of large policies. Pretrain on diverse data, then improve with RL on the specific robot. The objective is the RLHF one:

$\max_\theta\; \mathbb{E}_{a\sim\pi_\theta}\left[r_\psi(s,a)\right] - \beta\, D_{\text{KL}}\!\left(\pi_\theta \,\|\, \pi_{\text{ref}}\right),$

whose optimal solution is available in closed form,

$\pi^*(a\mid s) \propto \pi_{\text{ref}}(a\mid s)\,\exp\!\left(r_\psi(s,a)/\beta\right).$

use burn::prelude::*;
 
pub struct CtdePpo<B: Backend> {
    /// One actor per agent, each conditioned on that agent's observation only.
    actors: Vec<GaussianPolicy<B>>,
    /// A single centralized critic over the JOINT observation. Training-time
    /// only — it is discarded before deployment.
    critic: Critic<B>,
    n_agents: usize,
}
 
impl<B: AutodiffBackend> CtdePpo<B> {
    /// Execution: each agent acts on what it can actually see.
    pub fn act(&self, obs: &[Tensor<B, 2>]) -> Vec<Tensor<B, 2>> {
        obs.iter()
            .zip(&self.actors)
            .map(|(o, actor)| actor.sample(o.clone()).0)
            .collect()
    }
 
    /// Training: the value estimate may use the joint state, which makes it
    /// stationary from each agent's perspective even while the others learn.
    pub fn value(&self, joint_obs: Tensor<B, 2>) -> Tensor<B, 1> {
        self.critic.forward(joint_obs).squeeze(1)
    }
 
    /// Counterfactual advantage: how much better was agent i's action than its
    /// average alternative, holding the other agents fixed? This is Chapter 10's
    /// baseline argument, extended to assign team credit.
    pub fn counterfactual_advantage(
        &self,
        agent: usize,
        joint_obs: Tensor<B, 2>,
        joint_actions: &[Tensor<B, 2>],
        n_samples: usize,
    ) -> Tensor<B, 1> {
        let q_actual = self.joint_q(joint_obs.clone(), joint_actions);
 
        // Marginalize agent `agent` out by resampling its action.
        let mut baseline = q_actual.zeros_like();
        for _ in 0..n_samples {
            let mut counterfactual = joint_actions.to_vec();
            counterfactual[agent] = self.actors[agent]
                .sample(self.agent_obs(&joint_obs, agent))
                .0;
            baseline = baseline + self.joint_q(joint_obs.clone(), &counterfactual);
        }
        q_actual - baseline.div_scalar(n_samples as f64)
    }
}

21.5 What a decade actually bought

The most useful thing this chapter can do is compare the open problems Kober's survey listed in 2013 against those Tang's listed in 2024.

And the genuinely new problems, absent from the 2013 list:

Long-horizon tasks and skill discovery. Not prominent in 2013 because short-horizon tasks were still hard. Now the main obstacle to general-purpose robots, and Chapter 19 stated why.

Principled system design. Tang's survey is pointed about this: action spaces, reward terms and architectures are chosen by heuristic and expert taste, with few controlled comparisons. This book's insistence on classical baselines and reported units is a small response to that complaint.

Scaling multi-robot learning. Barely addressed in 2013; now blocked by the complexity result of §21.3.

21.6 Chapter bridge

The frontier is where other agents enter: humans who cannot be simulated, teams whose complexity is provably worse, and foundation models supplying priors nobody wrote.

One thing remains. This book has built twenty-one chapters of machinery and demonstrated each piece separately. Chapter 22 assembles all of it into a single project: specify a task, formalize the MDP with every choice justified against the evidence in Parts III and IV, build a randomized simulation, train teacher–student PPO, evaluate against the L0–L5 rubric with honest statistics, and ship a native binary alongside a browser demo. It is the chapter that turns a book into a template you can fork.

References