Reasoning ModelsOpen in Colab
Agentic project loop
Task definition·Context & tools·Orchestration loop·Evaluation·Guardrails·Operations feedback

Reasoning Models

Standard LLMs predict the next token given their context. They do a fixed amount of computation per token, which means a one-step multiplication and a twenty-step proof look equally hard from the model's perspective. Reasoning models break that constraint: they spend extra compute thinking before answering, and they are trained to exploit it. The result is dramatically better performance on tasks requiring multi-step logic, math, and code.

Real-world examples: OpenAI o1/o3, DeepSeek-R1, Google Gemini Flash Thinking, and Qwen QwQ. All are trained on top of strong base LLMs but with an additional reasoning-focused post-training stage.

How standard CoT differs from reasoning models

Chain-of-thought prompting asks a base model to produce intermediate steps before its final answer. This helps, but the model was not trained to produce those steps reliably — it just completes a pattern.

Reasoning models internalize CoT: they are explicitly trained to generate long reasoning traces before committing to an answer. The traces may be:

  • Visible (DeepSeek-R1 outputs <think>...</think> blocks before the answer)
  • Hidden (o1/o3 run an internal "thinking" pass that users don't see)

Either way, the model allocates more forward passes, and more tokens, to hard problems than to easy ones.

Same architecture, different training

Reasoning models use the same decoder-only Transformer architecture as base LLMs. The difference is entirely in how they are trained after pretraining: RL that rewards correct final answers over long reasoning chains.

Train-time compute scaling: RL on reasoning traces

The training recipe for reasoning models uses reinforcement learning to teach the model to reason:

  1. Generate several candidate reasoning chains + final answers for each problem.
  2. Verify the final answer against a ground-truth checker (math answer, unit-test pass, or a reward model for open-ended problems).
  3. Reinforce chains that reached the correct answer; penalize those that didn't.

The model learns (entirely through trial and error) that longer, more deliberate chains lead to better answers. No human-written reasoning traces are required; the reward signal comes from verifying the end result.

LRL=−Eτ∼πθ[R(τ)]where R(τ)=1[final answer is correct]\mathcal{L}_{\text{RL}} = -\mathbb{E}_{\tau \sim \pi_\theta}\big[R(\tau)\big] \quad \text{where } R(\tau) = \mathbf{1}[\text{final answer is correct}]

Popular algorithms for this step include GRPO (Group Relative Policy Optimization, used by DeepSeek) and PPO (used by OpenAI o1). Both keep the model close to a reference policy to prevent reward hacking.

Deep dive
GRPO: Group Relative Policy Optimization
Includes runnable notebook

Scaling law at train time: running RL for more steps lets the model discover more effective reasoning strategies. In the DeepSeek-R1 paper, model accuracy on competition math benchmarks rises near-monotonically with RL training duration — longer training = longer and better reasoning traces at inference.

Test-time compute scaling: budget forcing

Once a reasoning model is trained, you can control how much it thinks at inference time. The technique is called budget forcing: inject keywords or constraints that tell the model to think more or less.

"Wait" trick: appending the string "Wait" to the model's partial reasoning trace nudges it to pause and reconsider, often catching errors before committing. This can be done multiple times.

Text
[Problem] Prove that √2 is irrational.

[Model thinks...]
Assume √2 = p/q in lowest terms. Then 2 = p²/q², so p² = 2q²...
Wait, let me check the parity argument more carefully.
If p² is even then p must be even — write p = 2k...
[continues longer reasoning chain]
[Answer] √2 is irrational because any representation p/q leads to a contradiction.

Scaling curve: empirically, reasoning accuracy improves with more thinking tokens, but with diminishing returns. There is an optimal compute budget per problem difficulty; hard competition problems benefit from much longer chains than grade-school arithmetic.

Sampling more, and the verifier that decides what it is worth

Budget forcing spends compute inside one answer. The other way to spend it is across answers: sample NN attempts and pick the best. That makes the diminishing-returns story concrete, because the two halves separate cleanly.

What test-time compute actually buys

Sampling morestep 1 of 4
How often is a correct answer even in there
pass@N — 1 − (1−p)^N — is the chance that at least one of your N samples is right. It is what a perfect verifier would score, and a hard ceiling on any way of picking among the same samples.

A model right 35% of the time has a correct answer somewhere in 16 samples almost always. Whether you can find it is a different question.

Move N and watch the ceiling saturate long before the compute does.

the oracle ceilingbase model 35% · N = 16
0.250.500.751.001248163264128pass@N — the oracle ceilingaccuracyN — samples drawn per question
the ceiling

pass@16 = 0.999
a correct answer is in the batch this often

pass@N (oracle)

A model answers correctly with probability 0.35 on any single attempt. Sample N attempts, score each with a verifier (correct ~ N(1, σ), wrong ~ N(0, σ)) and return the best. pass@N is the oracle — what a perfect verifier would score, and a ceiling on any selection over the same samples. Every figure is simulated over thousands of trials here.

pass@N=1−(1−p)N\text{pass@}N = 1 - (1-p)^N is the chance a correct answer is somewhere in the batch — what a perfect verifier would score, and a hard ceiling on any way of choosing among the same samples. A model right 35% of the time has a correct answer in 16 samples 99.9% of the time. Whether you can find it is a separate problem, and it is the one that decides the payoff.

The verifier sets how much of the ceiling you collect. At N=16N = 16 a near-perfect verifier collects all 0.999 of it; one with noise 1.0 collects 75.9%; one with noise 3.0 collects 50.0%. The sampling is identical in all three cases.

So returns to more compute are non-monotone in verifier quality. Going from 16 to 128 samples (8× the spend) buys +0.001 with a near-perfect verifier (already saturated against the ceiling), +0.130 with a mediocre one, and +0.096 with a bad one. Both extremes waste the extra budget, for opposite reasons.

And that reframes the train-vs-test-time choice as a bet on selection. Best-of-16 on the 35% model with a good verifier scores 0.953 — equivalent to owning a 95.3%-single-shot model. With a poor verifier the same 16× spend scores 0.500. Same sampler, same compute, completely different substitution rate against simply training a better model. Measure your verifier against pass@N before budgeting for samples; that ratio, not the sample count, is what to improve.

Train-time vs test-time scaling

Train-timeTest-time
What is scaledRL training stepsTokens generated before answering
EffectModel learns better strategiesModel applies strategies more carefully
CostOne-time (training)Per-query (inference)
ControlFixed at deploymentAdjustable per request

DeepSeek-R1: a reference implementation

DeepSeek-R1 (2025) is the most transparent publicly documented reasoning model. Key facts:

  • Architecture: decoder-only Transformer (same class as LLaMA/GPT)
  • Training: pretraining → SFT → GRPO-based RL with verifiable reward signals
  • Reasoning traces: explicitly output between <think> and </think> tags; the final answer follows after </think>
  • Emergent behaviour: the RL stage produced spontaneous "aha moments" — the model learned to revisit and correct its own earlier steps without being told to
  • Open weights: DeepSeek released both the 671 B MoE model and smaller distilled variants (7 B, 14 B, 70 B parameters)
Text
Example output format:
<think>
  Let me break this into cases...
  Case 1: if n is even, then...
  Wait, that doesn't work for n=2. Let me reconsider.
  Actually, the key insight is...
</think>
The answer is 42, because [brief explanation].

When to use a reasoning model

Reasoning models are not universally better. They are slower and more expensive per query. Choose them when:

✅ Hard, verifiable tasks — math olympiad problems, competitive programming, formal proofs, scientific calculations where correctness is binary and checkable

✅ Multi-step planning — tasks requiring correct intermediate state (e.g. agentic coding that spans many files)

✅ Low latency tolerance — the user or system can wait several seconds or more for a better answer

❌ High-volume, latency-sensitive tasks — classification, extraction, summarisation of a single document: standard LLMs or even small fine-tuned models are cheaper and fast enough

❌ Tasks requiring real-time streaming — reasoning models often hold back output until thinking is complete

❌ Creative generation — longer thinking rarely helps prose, dialogue, or style

Thinking tokens cost money

A reasoning model might generate 2,000–10,000 thinking tokens before outputting 200 answer tokens. At typical API prices, this is 10–50× more expensive than a standard completion of the same length. Always measure whether the quality uplift is worth the cost for your specific task.

Exercise

What is the key difference between train-time compute scaling and test-time compute scaling for reasoning models?

Need a hint?

One is done once; the other happens per query.

Exercise

DeepSeek-R1 is trained with RL using a verifiable reward signal. What does 'verifiable' mean in this context, and why does it matter?

Need a hint?

Think about what the reward model is checking.

Exercise

Which task is the BEST fit for a reasoning model (like o3 or DeepSeek-R1) over a standard LLM?

Need a hint?

Reasoning models trade latency and cost for accuracy on hard, multi-step problems.

Common mistakes

  1. Using a reasoning model for simple tasks: calling o1 to classify a sentiment or extract a date is wasteful; a standard model or even a smaller fine-tuned model does it just as well for a fraction of the cost.
  2. Prompting like a base LLM: don't tell a reasoning model to "think step by step"; it already does. Adding CoT triggers can interfere with its internal reasoning strategy.
  3. Ignoring the thinking trace: for debugging, the <think> trace (when visible) is the best diagnostic. If the reasoning is sound but the answer is wrong, the issue is in the final-step extraction. If the reasoning itself is flawed, the problem is the model's knowledge or strategy.
  4. Equating reasoning length with quality: longer traces are generally better, but a model can also ramble. Evaluate the final answer, not the token count.

Key takeaways

  • Reasoning models are standard LLMs post-trained with RL on verifiable outcomes — they learn to generate useful CoT traces without explicit trace supervision.
  • Train-time scaling (more RL steps) and test-time scaling (more thinking tokens) both improve accuracy, and they compose.
  • Budget forcing (e.g. the "Wait" trick) is a simple way to give the model more test-time compute without retraining.
  • DeepSeek-R1 is the most documented open example: GRPO-based RL, <think> tags, emergent self-correction.
  • Use reasoning models for hard, verifiable, low-throughput tasks; fall back to standard LLMs for speed and cost.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Reasoning ModelsAttention & TransformersNatural Language Processi…Chain-of-Thought & Struct…Foundation Models & Scali…Preference Tuning: RLHF &…GRPO: Group Relative Poli…