Imitation Learning
Imitation Learning (IL) learns a policy from expert demonstrations, bypassing the need to define a reward function. Instead of specifying what good behavior looks like numerically, you show the agent, and it learns to imitate.
Applications: robotic manipulation from human teleoperation, autonomous driving from human drivers, game playing from professional replays, and (in a certain framing) RLHF for LLMs.
1. Behavioral Cloning (BC)
The simplest approach: treat imitation as supervised learning. Collect a dataset of (state, expert action) pairs, then train a policy to minimize:
For continuous actions, is MSE; for discrete actions, cross-entropy.
Problem — distribution shift: During training, the policy sees only states the expert visited. At test time, any small mistake moves the policy to a state not in the training set. From that out-of-distribution state, the policy makes another mistake, leading to further drift — errors compound.
The formal result: if the policy makes an error with probability per step, the total expected error over a trajectory of length is rather than , because each early mistake causes a cascade.
2. DAgger: Dataset Aggregation
DAgger (Ross et al., 2011) fixes distribution shift by iteratively collecting data from states the learned policy actually visits:
Initialize D = ∅, π₁ = any policy
for i = 1, 2, 3, ...:
1. Roll out πᵢ in the environment to collect states {s₁, s₂, ...}
2. Query expert for labels: aⱼ* = π_expert(sⱼ) for each visited state sⱼ
3. Aggregate: D ← D ∪ {(sⱼ, aⱼ*)}
4. Train πᵢ₊₁ via BC on the full dataset D
Key insight: By querying the expert at states the learned policy visits (not just states the expert visits), DAgger ensures the training distribution matches the test distribution. Under mild assumptions, DAgger achieves error — linear rather than quadratic.
Limitation: requires querying the expert at every training iteration, which is expensive if the expert is a human. Impractical when the expert is not queryable (e.g., learning from fixed offline video demonstrations).
Watch the loop run
Lane keeping with a PD expert and a nearest-neighbour clone. The demonstrations are excellent, the clone fits them almost perfectly, and it still leaves the road.
DAgger — why a better expert makes a worse clone
Three demonstrations from an expert PD controller, 120 (state, action) pairs in total. Look at the scale: the expert is competent, so it never let the car get further than y = 0.10 from the centre line, and the largest steering correction anywhere in the dataset is 0.84. Both of those numbers are about to matter more than the dataset's size.
Lane keeping with a PD expert and a nearest-neighbour clone. The demonstrations are excellent and the clone fits them almost perfectly, but the expert never let the car drift, so the largest correction anywhere in the data is 0.84 — and when a gust needs −7.20, the clone can only supply −0.38. Watch the aggregation loop manufacture the states the expert would never visit, then two measured payoffs: a steadier expert makes cloning monotonically worse across a 7-point sweep, and the held-out validation score misses the failure by 253×.
What to notice:
- Frame 2 vs frame 3. The clone's action error on held-out expert states is 0.014, and at a state the expert never visited it under-corrects by 19×. Both are true of the same model at the same time. The supervised score is not wrong, it is answering a different question.
- Frame 3: the structural bound. A clone can never output a correction larger than the largest one in its data, and the expert's competence is exactly what keeps that number small. The largest steering action anywhere in these demonstrations is 0.84; recovering from the gust needs 7.20.
- Frame 7: same model class, same , same code. Aggregation changes only the state distribution, and tracking cost falls 24×. Nothing was learned about control that behavioral cloning could not already represent.
- Frame 9: the payoff, and the counterintuitive one. Sweeping how steadily the expert drives, BC's cost gets monotonically worse as the expert gets better (105× across seven levels), while DAgger stays flat. This is why "collect more expert demonstrations" is the wrong instinct: more data from a good expert is more of the same sliver.
Two caveats the trace makes explicit rather than hiding. First, the bound above is worst-case: measured on this environment, cumulative deviation grows with exponent 0.44 for BC and 0.29 for DAgger, because both policies eventually stabilize — BC produces a large bounded transient here rather than a quadratic blow-up. What does hold is that BC's disadvantage widens with the horizon (1.8× at , 3.7× at ). Second, an earlier version of this trace used a linear learner against the linear expert and behavioral cloning never failed at all — when the hypothesis class is globally correct, extrapolation is exact and covariate shift costs nothing. Compounding error needs a learner that cannot extrapolate, which is every real policy network.
3. GAIL: Generative Adversarial Imitation Learning
GAIL (Ho & Ermon, 2016) reframes imitation as: find a policy whose state-action occupancy measure matches the expert's. It uses a GAN-style discriminator:
- Discriminator : trained to distinguish (state, action) pairs from the expert vs. from the current policy
- Policy : trained (with RL) to fool the discriminator — produce state-action pairs that look expert-like
The discriminator's output serves as an implicit reward signal — the policy gets rewarded for being indistinguishable from the expert.
Advantages over BC:
- Recovers more robust policies because it matches the full trajectory distribution, not just one-step imitation
- Works well even with limited expert data, since the discriminator generalizes
Disadvantages:
- Requires interacting with the real environment during training (to generate policy rollouts for the discriminator)
- Inherits GAN training instabilities
- Computationally expensive: alternates RL + discriminator updates
4. Inverse Reinforcement Learning
IL and Inverse RL (IRL) are related but different:
| Imitation Learning | Inverse RL | |
|---|---|---|
| Goal | Recover a policy that mimics the expert | Recover the reward function that the expert optimizes |
| Output | A policy | A reward function |
| Generalization | Policy may not generalize to new environments | Reward function generalizes if environment changes |
| Complexity | Simpler (direct supervision) | Harder (requires solving RL inner loop) |
5. Imitation Learning and RLHF
RLHF for LLMs uses a form of imitation learning:
- SFT (Supervised Fine-Tuning): behavioral cloning on high-quality human demonstrations
- Reward model: learn a reward function from human preference comparisons (inverse RL spirit)
- PPO: optimize the policy against the learned reward
The SFT stage is essentially behavioral cloning: train the LLM to imitate curated human responses. The reward model + PPO stage then corrects for the distribution shift problem by having the policy interact with the reward model and receive feedback.
Summary
| Method | Expert access needed | Distribution shift | Complexity |
|---|---|---|---|
| BC | Offline demos only | Severe ( error) | Low |
| DAgger | Online queries to expert | Eliminated ( error) | Medium |
| GAIL | Offline demos + env interaction | Avoided (matched occupancy) | High |
Related concepts
- From Policy Gradient to RLHF (PPO): SFT-as-behavioral-cloning and the reward-model step that follows
- Policy Gradient & Actor-Critic: the RL machinery GAIL optimizes against its discriminator reward
- Inverse Reinforcement Learning: recovering the reward instead of copying the policy