Transformers Quiz

Transformers Quiz

Check your understanding of the full course — the QKV computation and its √d scaling, multi-head attention and positional encodings, where a block's parameters live, the inference optimizations behind modern LLMs, the Chinchilla recipe, and sparse expert layers.

Exercise

In self-attention, the output for each token is a…

Need a hint?

softmax(QKᵀ/√d)V — read it right to left.

Exercise

Why divide QKᵀ by √d before the softmax?

Need a hint?

What's the variance of a dot product of two random d-dimensional vectors?

Exercise

Why use multiple attention heads instead of one big one?

Need a hint?

One softmax produces one weighting pattern per token.

Exercise

Why do transformers need positional encodings at all?

Need a hint?

Shuffle the input tokens — what changes in the attention computation?

Exercise

What is the transformer's key computational advantage over RNNs?

Need a hint?

What did sequential hidden-state updates prevent?

Exercise

In a standard Transformer layer (d_ff = 4·d_model), where do most of the parameters live?

Need a hint?

Compare 4·d² (attention) with 2·d·d_ff (FFN).

Exercise

Grouped-query attention (GQA) is used by LLaMA and Mistral primarily to reduce…

Need a hint?

What buffer grows with every generated token, per layer, at inference time?

Exercise

Under a fixed compute budget, the Chinchilla scaling result says you should train…

Need a hint?

Chinchilla-70B beat Gopher-280B at the same compute. How?

Exercise

Mixtral 8×7B has ~47B total parameters but runs each token through only ~13B. What makes this possible?

Need a hint?

How many of the 8 expert FFNs does the router pick per token?

Exercise

Self-attention assumes only permutation symmetry and re-injects order via positional encodings. In the ML project loop, that design is a change to which slot?

Need a hint?

It defines what relationships the model can represent — its architecture.

Exercise

Transfer test — a technique this course did NOT teach: a drift monitor that watches production error and triggers a retrain when it rises. Which slot of the ML project loop does it belong to?

Need a hint?

It operates after deployment, feeding signal back to the start of the loop.