Reinforcement Learning Quiz
Check your understanding of RL — the Markov property, the Bellman optimality target, why Q-learning is off-policy, when policy gradients win, why model rollouts stay short, and what PPO's clip actually does.
Exercise
The Markov property of an MDP states that…
Need a hint?
How much history does the transition function get to see?
Exercise
The Bellman optimality equation for Q* says Q*(s, a) equals…
Need a hint?
Immediate reward plus the discounted value of acting optimally afterward.
Exercise
Why is Q-learning called an off-policy algorithm?
Need a hint?
Compare the action it takes with the action in its update target.
Exercise
What does ε-greedy action selection accomplish?
Need a hint?
What happens to an agent that always exploits from day one?
Exercise
How do policy-gradient methods differ from value-based ones like Q-learning?
Need a hint?
What object does each method parameterize and optimize?
Exercise
Model-based RL (Dyna, MBPO) generates synthetic experience from a learned world model — but keeps the imagined rollouts short. Why?
Need a hint?
What happens to a 1%-per-step model error over 50 steps?
Exercise
In PPO's clipped objective, what happens to a token/action whose importance ratio ρ has already moved past 1+ε with positive advantage?
Need a hint?
min(ρA, clip(ρ, 1−ε, 1+ε)·A) — which branch wins, and what's its gradient in ρ?
Exercise
In RL, an agent must explore to gather useful experience, and there is no fixed dataset. Which slot of the ML project loop does exploration most directly correspond to?
Need a hint?
It determines what training examples the learner ever sees — but happening online.
Exercise
Transfer test — a technique this course did NOT teach: k-fold cross-validation to estimate how a model will generalize. Which slot of the ML project loop does it belong to?
Need a hint?
It's a way to measure generalization, not to train the model.