Reasoning Models
Standard LLMs predict the next token given their context. They do a fixed amount of computation per token, which means a one-step multiplication and a twenty-step proof look equally hard from the model's perspective. Reasoning models break that constraint: they spend extra compute thinking before answering, and they are trained to exploit it. The result is dramatically better performance on tasks requiring multi-step logic, math, and code.
Real-world examples: OpenAI o1/o3, DeepSeek-R1, Google Gemini Flash Thinking, and Qwen QwQ. All are trained on top of strong base LLMs but with an additional reasoning-focused post-training stage.
How standard CoT differs from reasoning models
Chain-of-thought prompting asks a base model to produce intermediate steps before its final answer. This helps, but the model was not trained to produce those steps reliably — it just completes a pattern.
Reasoning models internalize CoT: they are explicitly trained to generate long reasoning traces before committing to an answer. The traces may be:
- Visible (DeepSeek-R1 outputs
<think>...</think>blocks before the answer) - Hidden (o1/o3 run an internal "thinking" pass that users don't see)
Either way, the model allocates more forward passes, and more tokens, to hard problems than to easy ones.
Same architecture, different training
Reasoning models use the same decoder-only Transformer architecture as base LLMs. The difference is entirely in how they are trained after pretraining: RL that rewards correct final answers over long reasoning chains.
Train-time compute scaling: RL on reasoning traces
The training recipe for reasoning models uses reinforcement learning to teach the model to reason:
- Generate several candidate reasoning chains + final answers for each problem.
- Verify the final answer against a ground-truth checker (math answer, unit-test pass, or a reward model for open-ended problems).
- Reinforce chains that reached the correct answer; penalize those that didn't.
The model learns (entirely through trial and error) that longer, more deliberate chains lead to better answers. No human-written reasoning traces are required; the reward signal comes from verifying the end result.
Popular algorithms for this step include GRPO (Group Relative Policy Optimization, used by DeepSeek) and PPO (used by OpenAI o1). Both keep the model close to a reference policy to prevent reward hacking.
Scaling law at train time: running RL for more steps lets the model discover more effective reasoning strategies. In the DeepSeek-R1 paper, model accuracy on competition math benchmarks rises near-monotonically with RL training duration — longer training = longer and better reasoning traces at inference.
Test-time compute scaling: budget forcing
Once a reasoning model is trained, you can control how much it thinks at inference time. The technique is called budget forcing: inject keywords or constraints that tell the model to think more or less.
"Wait" trick: appending the string "Wait" to the model's partial reasoning trace nudges it to pause and reconsider, often catching errors before committing. This can be done multiple times.
[Problem] Prove that √2 is irrational.
[Model thinks...]
Assume √2 = p/q in lowest terms. Then 2 = p²/q², so p² = 2q²...
Wait, let me check the parity argument more carefully.
If p² is even then p must be even — write p = 2k...
[continues longer reasoning chain]
[Answer] √2 is irrational because any representation p/q leads to a contradiction.
Scaling curve: empirically, reasoning accuracy improves with more thinking tokens, but with diminishing returns. There is an optimal compute budget per problem difficulty; hard competition problems benefit from much longer chains than grade-school arithmetic.
Sampling more, and the verifier that decides what it is worth
Budget forcing spends compute inside one answer. The other way to spend it is across answers: sample attempts and pick the best. That makes the diminishing-returns story concrete, because the two halves separate cleanly.
What test-time compute actually buys
How often is a correct answer even in there
A model right 35% of the time has a correct answer somewhere in 16 samples almost always. Whether you can find it is a different question.
Move N and watch the ceiling saturate long before the compute does.
the ceiling
pass@16 = 0.999
a correct answer is in the batch this often
A model answers correctly with probability 0.35 on any single attempt. Sample N attempts, score each with a verifier (correct ~ N(1, σ), wrong ~ N(0, σ)) and return the best. pass@N is the oracle — what a perfect verifier would score, and a ceiling on any selection over the same samples. Every figure is simulated over thousands of trials here.
is the chance a correct answer is somewhere in the batch — what a perfect verifier would score, and a hard ceiling on any way of choosing among the same samples. A model right 35% of the time has a correct answer in 16 samples 99.9% of the time. Whether you can find it is a separate problem, and it is the one that decides the payoff.
The verifier sets how much of the ceiling you collect. At a near-perfect verifier collects all 0.999 of it; one with noise 1.0 collects 75.9%; one with noise 3.0 collects 50.0%. The sampling is identical in all three cases.
So returns to more compute are non-monotone in verifier quality. Going from 16 to 128 samples (8× the spend) buys +0.001 with a near-perfect verifier (already saturated against the ceiling), +0.130 with a mediocre one, and +0.096 with a bad one. Both extremes waste the extra budget, for opposite reasons.
And that reframes the train-vs-test-time choice as a bet on selection. Best-of-16 on the 35% model with a good verifier scores 0.953 — equivalent to owning a 95.3%-single-shot model. With a poor verifier the same 16× spend scores 0.500. Same sampler, same compute, completely different substitution rate against simply training a better model. Measure your verifier against pass@N before budgeting for samples; that ratio, not the sample count, is what to improve.
Train-time vs test-time scaling
| Train-time | Test-time | |
|---|---|---|
| What is scaled | RL training steps | Tokens generated before answering |
| Effect | Model learns better strategies | Model applies strategies more carefully |
| Cost | One-time (training) | Per-query (inference) |
| Control | Fixed at deployment | Adjustable per request |
DeepSeek-R1: a reference implementation
DeepSeek-R1 (2025) is the most transparent publicly documented reasoning model. Key facts:
- Architecture: decoder-only Transformer (same class as LLaMA/GPT)
- Training: pretraining → SFT → GRPO-based RL with verifiable reward signals
- Reasoning traces: explicitly output between
<think>and</think>tags; the final answer follows after</think> - Emergent behaviour: the RL stage produced spontaneous "aha moments" — the model learned to revisit and correct its own earlier steps without being told to
- Open weights: DeepSeek released both the 671 B MoE model and smaller distilled variants (7 B, 14 B, 70 B parameters)
Example output format:
<think>
Let me break this into cases...
Case 1: if n is even, then...
Wait, that doesn't work for n=2. Let me reconsider.
Actually, the key insight is...
</think>
The answer is 42, because [brief explanation].
When to use a reasoning model
Reasoning models are not universally better. They are slower and more expensive per query. Choose them when:
✅ Hard, verifiable tasks — math olympiad problems, competitive programming, formal proofs, scientific calculations where correctness is binary and checkable
✅ Multi-step planning — tasks requiring correct intermediate state (e.g. agentic coding that spans many files)
✅ Low latency tolerance — the user or system can wait several seconds or more for a better answer
❌ High-volume, latency-sensitive tasks — classification, extraction, summarisation of a single document: standard LLMs or even small fine-tuned models are cheaper and fast enough
❌ Tasks requiring real-time streaming — reasoning models often hold back output until thinking is complete
❌ Creative generation — longer thinking rarely helps prose, dialogue, or style
Thinking tokens cost money
A reasoning model might generate 2,000–10,000 thinking tokens before outputting 200 answer tokens. At typical API prices, this is 10–50× more expensive than a standard completion of the same length. Always measure whether the quality uplift is worth the cost for your specific task.
Exercise
What is the key difference between train-time compute scaling and test-time compute scaling for reasoning models?
Need a hint?
One is done once; the other happens per query.
Exercise
DeepSeek-R1 is trained with RL using a verifiable reward signal. What does 'verifiable' mean in this context, and why does it matter?
Need a hint?
Think about what the reward model is checking.
Exercise
Which task is the BEST fit for a reasoning model (like o3 or DeepSeek-R1) over a standard LLM?
Need a hint?
Reasoning models trade latency and cost for accuracy on hard, multi-step problems.
Common mistakes
- Using a reasoning model for simple tasks: calling o1 to classify a sentiment or extract a date is wasteful; a standard model or even a smaller fine-tuned model does it just as well for a fraction of the cost.
- Prompting like a base LLM: don't tell a reasoning model to "think step by step"; it already does. Adding CoT triggers can interfere with its internal reasoning strategy.
- Ignoring the thinking trace: for debugging, the
<think>trace (when visible) is the best diagnostic. If the reasoning is sound but the answer is wrong, the issue is in the final-step extraction. If the reasoning itself is flawed, the problem is the model's knowledge or strategy. - Equating reasoning length with quality: longer traces are generally better, but a model can also ramble. Evaluate the final answer, not the token count.
Key takeaways
- Reasoning models are standard LLMs post-trained with RL on verifiable outcomes — they learn to generate useful CoT traces without explicit trace supervision.
- Train-time scaling (more RL steps) and test-time scaling (more thinking tokens) both improve accuracy, and they compose.
- Budget forcing (e.g. the "Wait" trick) is a simple way to give the model more test-time compute without retraining.
- DeepSeek-R1 is the most documented open example: GRPO-based RL,
<think>tags, emergent self-correction. - Use reasoning models for hard, verifiable, low-throughput tasks; fall back to standard LLMs for speed and cost.
Related concepts
- Chain-of-Thought & Structured Output: the prompting precursor to trained reasoning
- Foundation Models & Scaling Laws: the pretraining that provides the base capability reasoning RL refines
- Preference Tuning: RLHF & DPO: the RL machinery reasoning training adapts for verifiable rewards