Transformers Quiz
Check your understanding of the full course — the QKV computation and its √d scaling, multi-head attention and positional encodings, where a block's parameters live, the inference optimizations behind modern LLMs, the Chinchilla recipe, and sparse expert layers.
Exercise
In self-attention, the output for each token is a…
Need a hint?
softmax(QKᵀ/√d)V — read it right to left.
Exercise
Why divide QKᵀ by √d before the softmax?
Need a hint?
What's the variance of a dot product of two random d-dimensional vectors?
Exercise
Why use multiple attention heads instead of one big one?
Need a hint?
One softmax produces one weighting pattern per token.
Exercise
Why do transformers need positional encodings at all?
Need a hint?
Shuffle the input tokens — what changes in the attention computation?
Exercise
What is the transformer's key computational advantage over RNNs?
Need a hint?
What did sequential hidden-state updates prevent?
Exercise
In a standard Transformer layer (d_ff = 4·d_model), where do most of the parameters live?
Need a hint?
Compare 4·d² (attention) with 2·d·d_ff (FFN).
Exercise
Grouped-query attention (GQA) is used by LLaMA and Mistral primarily to reduce…
Need a hint?
What buffer grows with every generated token, per layer, at inference time?
Exercise
Under a fixed compute budget, the Chinchilla scaling result says you should train…
Need a hint?
Chinchilla-70B beat Gopher-280B at the same compute. How?
Exercise
Mixtral 8×7B has ~47B total parameters but runs each token through only ~13B. What makes this possible?
Need a hint?
How many of the 8 expert FFNs does the router pick per token?
Exercise
Self-attention assumes only permutation symmetry and re-injects order via positional encodings. In the ML project loop, that design is a change to which slot?
Need a hint?
It defines what relationships the model can represent — its architecture.
Exercise
Transfer test — a technique this course did NOT teach: a drift monitor that watches production error and triggers a retrain when it rises. Which slot of the ML project loop does it belong to?
Need a hint?
It operates after deployment, feeding signal back to the start of the loop.