Reinforcement Learning Quiz

Reinforcement Learning Quiz

Check your understanding of RL — the Markov property, the Bellman optimality target, why Q-learning is off-policy, when policy gradients win, why model rollouts stay short, and what PPO's clip actually does.

Exercise

The Markov property of an MDP states that…

Need a hint?

How much history does the transition function get to see?

Exercise

The Bellman optimality equation for Q* says Q*(s, a) equals…

Need a hint?

Immediate reward plus the discounted value of acting optimally afterward.

Exercise

Why is Q-learning called an off-policy algorithm?

Need a hint?

Compare the action it takes with the action in its update target.

Exercise

What does ε-greedy action selection accomplish?

Need a hint?

What happens to an agent that always exploits from day one?

Exercise

How do policy-gradient methods differ from value-based ones like Q-learning?

Need a hint?

What object does each method parameterize and optimize?

Exercise

Model-based RL (Dyna, MBPO) generates synthetic experience from a learned world model — but keeps the imagined rollouts short. Why?

Need a hint?

What happens to a 1%-per-step model error over 50 steps?

Exercise

In PPO's clipped objective, what happens to a token/action whose importance ratio ρ has already moved past 1+ε with positive advantage?

Need a hint?

min(ρA, clip(ρ, 1−ε, 1+ε)·A) — which branch wins, and what's its gradient in ρ?

Exercise

In RL, an agent must explore to gather useful experience, and there is no fixed dataset. Which slot of the ML project loop does exploration most directly correspond to?

Need a hint?

It determines what training examples the learner ever sees — but happening online.

Exercise

Transfer test — a technique this course did NOT teach: k-fold cross-validation to estimate how a model will generalize. Which slot of the ML project loop does it belong to?

Need a hint?

It's a way to measure generalization, not to train the model.