Imitation LearningOpen in Colab

Imitation Learning

Imitation Learning (IL) learns a policy from expert demonstrations, bypassing the need to define a reward function. Instead of specifying what good behavior looks like numerically, you show the agent, and it learns to imitate.

Applications: robotic manipulation from human teleoperation, autonomous driving from human drivers, game playing from professional replays, and (in a certain framing) RLHF for LLMs.

1. Behavioral Cloning (BC)

The simplest approach: treat imitation as supervised learning. Collect a dataset D={(si,ai∗)}\mathcal{D} = \{(s_i, a_i^*)\} of (state, expert action) pairs, then train a policy πθ\pi_\theta to minimize:

L(θ)=E(s,a∗)∼D ⁣[ℓ(πθ(s), a∗)]\mathcal{L}(\theta) = \mathbb{E}_{(s,a^*) \sim \mathcal{D}}\!\left[\ell(\pi_\theta(s),\, a^*)\right]

For continuous actions, ℓ\ell is MSE; for discrete actions, cross-entropy.

Problem — distribution shift: During training, the policy sees only states the expert visited. At test time, any small mistake moves the policy to a state not in the training set. From that out-of-distribution state, the policy makes another mistake, leading to further drift — errors compound.

The formal result: if the policy makes an error with probability ϵ\epsilon per step, the total expected error over a trajectory of length TT is O(ϵT2)O(\epsilon T^2) rather than O(ϵT)O(\epsilon T), because each early mistake causes a cascade.

2. DAgger: Dataset Aggregation

DAgger (Ross et al., 2011) fixes distribution shift by iteratively collecting data from states the learned policy actually visits:

Code
Initialize D = ∅, π₁ = any policy
for i = 1, 2, 3, ...:
  1. Roll out πᵢ in the environment to collect states {s₁, s₂, ...}
  2. Query expert for labels: aⱼ* = π_expert(sⱼ) for each visited state sⱼ
  3. Aggregate: D ← D ∪ {(sⱼ, aⱼ*)}
  4. Train πᵢ₊₁ via BC on the full dataset D

Key insight: By querying the expert at states the learned policy visits (not just states the expert visits), DAgger ensures the training distribution matches the test distribution. Under mild assumptions, DAgger achieves O(ϵT)O(\epsilon T) error — linear rather than quadratic.

Limitation: requires querying the expert at every training iteration, which is expensive if the expert is a human. Impractical when the expert is not queryable (e.g., learning from fixed offline video demonstrations).

Watch the loop run

Lane keeping with a PD expert and a nearest-neighbour clone. The demonstrations are excellent, the clone fits them almost perfectly, and it still leaves the road.

DAgger — why a better expert makes a worse clone

1/10

Three demonstrations from an expert PD controller, 120 (state, action) pairs in total. Look at the scale: the expert is competent, so it never let the car get further than y = 0.10 from the centre line, and the largest steering correction anywhere in the dataset is 0.84. Both of those numbers are about to matter more than the dataset's size.

python
1D = expert_demos()
2pi = fit(D)
3
4for i in range(1, n_iters + 1):
5
6 # 1. roll the LEARNER, not the expert
7 S = states_visited(pi)
8
9 # 2. ask the expert what it would
10 # have done at each of those
11 A = [expert(s) for s in S]
12
13 # 3. aggregate: never discard
14 D = D + list(zip(S, A))
15
16 # 4. refit on everything so far
17 pi = fit(D)
state
demonstrated states (120 pairs)
-3.63.66-6lateral offset yvelocity v
what the expert showed us
pairs → 120max |y| → 0.10max |a| → 0.84lane edge → |y| = 1.5

Lane keeping with a PD expert and a nearest-neighbour clone. The demonstrations are excellent and the clone fits them almost perfectly, but the expert never let the car drift, so the largest correction anywhere in the data is 0.84 — and when a gust needs −7.20, the clone can only supply −0.38. Watch the aggregation loop manufacture the states the expert would never visit, then two measured payoffs: a steadier expert makes cloning monotonically worse across a 7-point sweep, and the held-out validation score misses the failure by 253×.

What to notice:

  • Frame 2 vs frame 3. The clone's action error on held-out expert states is 0.014, and at a state the expert never visited it under-corrects by 19×. Both are true of the same model at the same time. The supervised score is not wrong, it is answering a different question.
  • Frame 3: the structural bound. A clone can never output a correction larger than the largest one in its data, and the expert's competence is exactly what keeps that number small. The largest steering action anywhere in these demonstrations is 0.84; recovering from the gust needs 7.20.
  • Frame 7: same model class, same kk, same code. Aggregation changes only the state distribution, and tracking cost falls 24×. Nothing was learned about control that behavioral cloning could not already represent.
  • Frame 9: the payoff, and the counterintuitive one. Sweeping how steadily the expert drives, BC's cost gets monotonically worse as the expert gets better (105× across seven levels), while DAgger stays flat. This is why "collect more expert demonstrations" is the wrong instinct: more data from a good expert is more of the same sliver.

Two caveats the trace makes explicit rather than hiding. First, the O(ϵT2)O(\epsilon T^2) bound above is worst-case: measured on this environment, cumulative deviation grows with exponent 0.44 for BC and 0.29 for DAgger, because both policies eventually stabilize — BC produces a large bounded transient here rather than a quadratic blow-up. What does hold is that BC's disadvantage widens with the horizon (1.8× at T=10T=10, 3.7× at T=200T=200). Second, an earlier version of this trace used a linear learner against the linear expert and behavioral cloning never failed at all — when the hypothesis class is globally correct, extrapolation is exact and covariate shift costs nothing. Compounding error needs a learner that cannot extrapolate, which is every real policy network.

3. GAIL: Generative Adversarial Imitation Learning

GAIL (Ho & Ermon, 2016) reframes imitation as: find a policy whose state-action occupancy measure matches the expert's. It uses a GAN-style discriminator:

  • Discriminator Dϕ(s,a)D_\phi(s,a): trained to distinguish (state, action) pairs from the expert vs. from the current policy
  • Policy πθ\pi_\theta: trained (with RL) to fool the discriminator — produce state-action pairs that look expert-like
min⁡θmax⁡ϕ  Eπθ[log⁡Dϕ(s,a)]+Eπ∗[log⁡(1−Dϕ(s,a))]\min_\theta \max_\phi \;\mathbb{E}_{\pi_\theta}[\log D_\phi(s,a)] + \mathbb{E}_{\pi^*}[\log(1 - D_\phi(s,a))]

The discriminator's output −log⁡Dϕ(s,a)-\log D_\phi(s,a) serves as an implicit reward signal — the policy gets rewarded for being indistinguishable from the expert.

Advantages over BC:

  • Recovers more robust policies because it matches the full trajectory distribution, not just one-step imitation
  • Works well even with limited expert data, since the discriminator generalizes

Disadvantages:

  • Requires interacting with the real environment during training (to generate policy rollouts for the discriminator)
  • Inherits GAN training instabilities
  • Computationally expensive: alternates RL + discriminator updates

4. Inverse Reinforcement Learning

IL and Inverse RL (IRL) are related but different:

Imitation LearningInverse RL
GoalRecover a policy π\pi that mimics the expertRecover the reward function RR that the expert optimizes
OutputA policyA reward function
GeneralizationPolicy may not generalize to new environmentsReward function generalizes if environment changes
ComplexitySimpler (direct supervision)Harder (requires solving RL inner loop)

5. Imitation Learning and RLHF

RLHF for LLMs uses a form of imitation learning:

  1. SFT (Supervised Fine-Tuning): behavioral cloning on high-quality human demonstrations
  2. Reward model: learn a reward function from human preference comparisons (inverse RL spirit)
  3. PPO: optimize the policy against the learned reward

The SFT stage is essentially behavioral cloning: train the LLM to imitate curated human responses. The reward model + PPO stage then corrects for the distribution shift problem by having the policy interact with the reward model and receive feedback.

Summary

MethodExpert access neededDistribution shiftComplexity
BCOffline demos onlySevere (O(T2)O(T^2) error)Low
DAggerOnline queries to expertEliminated (O(T)O(T) error)Medium
GAILOffline demos + env interactionAvoided (matched occupancy)High

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Imitation LearningInverse Reinforcement Lea…Policy Gradient & Actor-C…From Policy Gradient to R…