Foundation Models & Scaling LawsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Foundation Models & Scaling Laws

A foundation model is a single, large, self-supervised model that is pretrained once on enormous amounts of text and then adapted (by prompting, retrieval, fine-tuning, or alignment) to thousands of downstream tasks. GPT-4-class systems, Claude, Gemini, and the open-weights Llama-3 family are all foundation models. They all share the same basic recipe: a decoder-only Transformer trained to predict the next token on a trillion-scale text corpus, followed by a post-training stage that turns the raw "internet autocomplete" into an instruction-following assistant.

This lesson covers the part of the recipe that is not architecture: the pretraining objective, the scale at which it is done, the scaling laws that make that scale predictable, what the context window actually costs, and a short forward-link to post-training (SFT → RM → RLHF/DPO), which gets its own course.

Real-world examples: GPT-4 / GPT-4o (OpenAI), Claude 3.5 Sonnet (Anthropic), Gemini 1.5 (Google DeepMind), Llama 3.1 (Meta, weights open), Mistral Large, and Qwen 2.5.

The pretraining objective: causal language modelling

Every modern decoder-only LLM is pretrained with the same single objective: at every position, predict the next token given the prefix so far. The loss is the average cross-entropy of the model's predicted distribution against the true token:

L(θ)=−∑t=1Tlog⁡pθ(xt∣x<t)\mathcal{L}(\theta) = -\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t})

This is causal language modelling (CLM): position tt is allowed to attend only to positions <t<t. The causal mask in the decoder's self-attention is what enforces this — without it, the model could see token xtx_t when predicting xtx_t and the objective would collapse to "copy the answer". That information leak is the only thing preventing a degenerate solution.

Concrete loss value. Cross-entropy is in nats per token. For a model whose predicted probability of the true next token is on average 0.180.18, the per-token loss is −ln⁡0.18≈1.71-\ln 0.18 \approx 1.71 nats. A token-level loss of about 1.91.9 on web text corresponds to a perplexity e1.9≈6.7e^{1.9} \approx 6.7 — the model behaves as if it had to pick between 6–7 equally likely options at every step.

Why causal LM and not masked LM (BERT)? Encoder-only models like BERT use a masked language modelling objective: replace 15 % of tokens with [MASK] and predict them from bidirectional context. That works great for embeddings and classification but is awkward for generation, because at inference time the model doesn't see masks. It has to produce tokens one at a time. Causal LM matches the inference pattern exactly: every training example is also a valid generation step. That alignment between training and inference is the deepest reason GPT-style models won the generative race.

Why next-token prediction is enough

Next-token prediction looks like a narrow objective, but it forces the model to learn syntax, world knowledge, arithmetic, code, reasoning chains, and dialogue conventions, because every one of those skills shows up somewhere in the training data, and you cannot predict the next token without modelling them. "Just predict the next token" turns out to be a remarkably general task.

Training data scale: the trillion-token regime

The training corpus for a frontier model is dominated by web text (Common Crawl, GitHub, books, papers, Wikipedia) heavily filtered and deduplicated. The unit of accounting is the token (sub-word piece). For reference:

ModelParameters NNTraining tokens DDD/ND / N ratio
GPT-3 (2020)175 B0.30 T∼\sim1.7
Gopher (2021)280 B0.30 T∼\sim1.1
Chinchilla (2022)70 B1.4 T20
Llama-3 8B (2024)8 B15 T∼\sim1900
Llama-3 70B (2024)70 B15 T∼\sim215

Note how the D/ND/N ratio jumps after the Chinchilla paper showed that GPT-3 and Gopher were severely under-trained for their size. The current frontier ratios (hundreds, even thousands of tokens per parameter) are well past Chinchilla-optimal, deliberately so — extra tokens are cheap at inference time and over-trained models pay for themselves in lower serving cost.

Scaling laws: Kaplan and Chinchilla

The empirical regularity that drives the whole field is that loss is a smooth, predictable power-law function of model size, dataset size, and compute — over many orders of magnitude.

Kaplan et al., 2020. OpenAI's original scaling laws paper fit independent power laws in NN (parameters), DD (tokens), and CC (compute), each with diminishing returns:

L(N)≈ANα,L(D)≈BDβ,L(C)≈KCγ.L(N) \approx \frac{A}{N^{\alpha}}, \qquad L(D) \approx \frac{B}{D^{\beta}}, \qquad L(C) \approx \frac{K}{C^{\gamma}}.

Kaplan's recommendation was to scale parameters faster than data — i.e. spend most of a compute budget on bigger models. GPT-3 was sized accordingly.

Hoffmann et al. ("Chinchilla"), 2022. A more careful joint fit gave a different recipe. The two-term form is:

L(N,D)≈ANα+BDβ+EL(N, D) \approx \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} + E

with rough exponents α≈0.34\alpha \approx 0.34 and β≈0.28\beta \approx 0.28, and an irreducible loss EE that the data itself imposes (text has entropy you cannot model away). The key result is that under a fixed compute budget C=6⋅N⋅DC = 6 \cdot N \cdot D FLOPs, the compute-optimal allocation is

D∗≈20⋅N∗(equivalently N∗=C/120,  D∗=20N∗).D^* \approx 20 \cdot N^* \qquad \text{(equivalently } N^* = \sqrt{C/120},\; D^* = 20 N^*\text{)}.

In words: spend roughly twenty training tokens per parameter. Earlier models were too big for their data budget. The "Chinchilla 70B" model trained on 1.4 T tokens out-performed the much larger Gopher 280B trained on 0.3 T at the same compute, vindicating the recipe.

Chinchilla in one line

For a fixed compute budget, the loss is minimised when every parameter sees about 20 training tokens. Bigger models train slower; smaller models with more data are cheaper to serve. Hence the modern preference for "over-trained" 8B/70B models with hundreds of tokens per parameter — the inference savings dwarf the small extra training cost.

Chinchilla scaling law: loss vs compute (log-log)

1e181e201e221e241e26Compute C = 6·N·D (FLOPs, log)235102050Loss L (log)1B params7B params70B paramsbudget CD = 20·N
compute C3.2e22 FLOPs
optimal N*16.2B
optimal D*325B tok
predicted loss2.07

Each curve sweeps training tokens D for a fixed model size N, plotting predicted loss L(N, D) against compute C = 6ND. The compute-optimal point per curve (★) follows the Chinchilla rule D ≈ 20·N. Slide the compute budget to see the matching optimal (N*, D*).

Why a sum of two power laws? Each term in L(N,D)=A/Nα+B/Dβ+EL(N, D) = A/N^{\alpha} + B/D^{\beta} + E captures a separate bottleneck: A/NαA/N^{\alpha} is the model-capacity floor (a tiny model cannot fit complex regularities no matter how much data you show it), B/DβB/D^{\beta} is the data floor (you cannot learn what you have never seen), and EE is the irreducible noise. At any fixed CC, the optimum balances the first two terms, and the algebra gives D∝ND \propto N with the constant set by the exponents, which works out to ≈20\approx 20 for the fitted values.

The context window and what it really costs

The context window is the maximum number of tokens LL the model can attend to in one forward pass. For vanilla self-attention, both compute and memory grow as

cost(L)=O(L2)⋅dmodel\text{cost}(L) = O(L^2) \cdot d_{\text{model}}

per layer — the score matrix QK⊤QK^\top is L×LL \times L. Doubling the context quadruples the cost. For L=128,000L = 128{,}000 tokens (Claude's standard window) this is 128,0002=1.64×1010128{,}000^2 = 1.64 \times 10^{10} attention scores per layer per head — billions, in a single forward step.

Two families of techniques attack this:

  1. Memory-efficient exact attention: FlashAttention tiles the computation so the full L×LL \times L matrix is never written to HBM. Compute is still O(L2)O(L^2), but the memory wall is broken and wall-clock time drops 2–4×.
  2. Sub-quadratic approximations: sliding-window attention, sparse global tokens (Longformer, BigBird), and linear-attention variants (Mamba/SSMs, Linformer) all trade some quality for asymptotically lower cost. These are how 1 M-token and 10 M-token windows are achieved in practice.

A third axis is position generalisation. RoPE-style positional encodings, with techniques such as YaRN, NTK-aware scaling, or position-interpolation, let a model trained at 4 K tokens be extended to 32 K or 128 K with only light fine-tuning. Without such tricks the model's positional understanding simply degrades past its training length.

What 'context window' really costs

A "1 M-token context" headline does not mean attention is free at that length. It means the engineering team paid the O(L2)O(L^2) cost with FlashAttention plus sparse/linear approximations, plus a position-encoding trick to keep the embeddings sane. Quality on long-context tasks degrades smoothly with LL and is best measured directly (needle-in-a-haystack, long-doc QA), not assumed from the advertised window.

Post-training, in one paragraph

A freshly pretrained model is a fluent autocomplete — it produces plausible web text. Turning it into an assistant requires three short additional training phases, collectively called post-training (or alignment):

  1. Supervised Fine-Tuning (SFT) on curated (prompt, response) pairs teaches the model the shape of a helpful answer — see Supervised Fine-Tuning.
  2. Reward Modelling (RM) trains a separate scalar head on pairwise preferences (chosen vs rejected responses).
  3. Preference optimisation: classical RLHF with PPO (uses the reward model + a KL leash) or DPO (a closed-form, RM-free alternative).

Post-training uses orders of magnitude less compute than pretraining (typically a few percent) but is responsible for the dramatic quality jump from "raw GPT-3 base model" to "ChatGPT". It is also where most of the visible behavioural differences between frontier models live; the underlying pretrained Transformers are remarkably similar.

Exercise

Under the Chinchilla scaling law, what does the compute-optimal recipe say about how to allocate a fixed training compute budget $C = 6ND$ between model size $N$ and tokens $D$?

Need a hint?

Twenty of something per one of something else.

Exercise

For vanilla self-attention in a Transformer, how does the compute cost scale with sequence length $L$, and what is the role of FlashAttention?

Need a hint?

Think about the score matrix $QK^\top$ — what shape is it?

Exercise

You have a training compute budget of $C = 1.2 \times 10^{23}$ FLOPs. Using the Chinchilla rule $N^* = \sqrt{C / 120}$, what is the compute-optimal model size $N^*$ (closest answer)?

Need a hint?

Plug in: $N^* = \sqrt{1.2 \times 10^{23} / 120} = \sqrt{10^{21}}$.

Common mistakes

  1. Treating D/ND/N as a hyperparameter to tune by intuition. It is not — Chinchilla's D≈20 ND \approx 20\,N is an empirical optimum under a fixed compute constraint. If you instead fix inference cost, you should deliberately over-train (more tokens per parameter) because cheap inference is worth the extra training tokens.
  2. Confusing parameters with FLOPs. Compute is roughly C≈6⋅N⋅DC \approx 6 \cdot N \cdot D for a dense Transformer (forward + backward + activations). A "1 T-parameter MoE" with only 100 B active per token uses much less compute than its parameter count suggests.
  3. Reading scaling laws as deterministic forecasts. They predict loss within a regime; capability "jumps" on downstream benchmarks at certain scales are not captured by smooth power laws.
  4. Assuming bigger context = better. Attention quality on long contexts is a separate empirical question; a 128 K window doesn't mean uniform recall across all 128 K tokens.
  5. Skipping the irreducible loss EE. It is real and non-zero. Power-law fits that omit it predict zero loss at infinite scale, which is nonsense — text has finite intrinsic entropy.

Key takeaways

  • Causal language modelling (next-token cross-entropy with a triangular mask) is the universal pretraining objective for modern LLMs; it matches inference exactly and forces the model to learn everything a generator needs.
  • Training data scale is now in the trillions of tokens; modern open models train at hundreds to thousands of tokens per parameter — well past Chinchilla-optimal, on purpose, to make inference cheap.
  • Chinchilla scaling law: L(N,D)=A/Nα+B/Dβ+EL(N, D) = A/N^{\alpha} + B/D^{\beta} + E with α≈0.34\alpha \approx 0.34, β≈0.28\beta \approx 0.28. The compute-optimal recipe is D∗≈20 N∗D^* \approx 20\,N^*, with N∗=C/120N^* = \sqrt{C/120} for budget C=6NDC = 6ND FLOPs.
  • Context window costs O(L2)O(L^2) for vanilla attention; FlashAttention removes the memory wall, sparse and linear variants reduce the compute, and RoPE-scaling tricks extend trained windows.
  • Post-training (SFT → RM → RLHF or DPO) is the small but decisive stage that turns a pretrained autocomplete into a usable assistant — covered in the Fine-Tuning & Alignment course.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Foundation Models & Scaling LawsRecurrent Neural NetworksCalculus for MLThe Transformer Architect…Modern Attention Mechanis…Supervised Fine-Tuning & …Preference Tuning: RLHF &…Mixture of ExpertsLLM Evaluation