Model Evaluation Quiz
Check your understanding: computing precision from a confusion matrix, spotting data leakage, rescaling a c-statistic into a Gini, and knowing when Bayesian optimization beats grid search.
Exercise
A confusion matrix shows: TP=80, FP=20, FN=40, TN=860. What is the precision of the positive class?
Need a hint?
Precision = TP / (TP + FP).
Exercise
In k-fold cross-validation, which estimate is a nearly unbiased estimate of the model's generalization error?
Need a hint?
Think about what 'held-out' data the model never saw during that fold's training.
Exercise
A fraud detection model is trained on transactions from Jan–Jun and tested on Jul–Dec. The feature 'days_since_last_fraud_in_account' is computed using the entire dataset. What type of data leakage is this?
Need a hint?
The feature uses future information to compute a 'past' feature for earlier transactions.
Exercise
Bayesian hyperparameter optimization outperforms random search for expensive objectives. What does it use to guide the search?
Need a hint?
It models the unknown objective function probabilistically.
Exercise
In image classification, random horizontal flipping, cropping, and color jitter are applied during training but not at test time. What regularization effect does this provide?
Need a hint?
The model sees different transformed versions of each image — what does this increase?
Exercise
Model A (256k-token vocabulary) reports lower perplexity than Model B (32k vocabulary) on the same corpus. What can you conclude?
Need a hint?
What does per-token cross-entropy depend on besides the model?
Exercise
You apply temperature scaling (dividing all logits by a tuned T > 1) to an overconfident classifier. What happens to its top-1 accuracy?
Need a hint?
Is dividing every logit by the same positive constant monotonic?
Exercise
A neural network reaches zero training error, and you keep growing it further. According to double descent, what can happen to test error?
Need a hint?
What happens past the interpolation threshold?
Exercise
A scorecard reports an AUC (c-statistic) of 0.72. What Gini coefficient — equivalently Somers' D — does the validation pack report?
Need a hint?
The two are rescalings of each other, with a coin flip sitting at 0 instead of 0.5.
Exercise
Team A reports KS = 0.41 from a ten-row decile table. Team B reports KS = 0.44 computed over every distinct score on the same predictions. Which is right?
Need a hint?
What cut points can each calculation search over?
Exercise
This whole course — metrics, validation, calibration, learning theory — is the deep treatment of which single slot of the ML project loop?
Need a hint?
It's the slot that decides whether the other five actually worked.
Exercise
Transfer test — a technique this course did NOT teach: a mixture-of-experts layer that routes each token to a few specialized sub-networks. Which slot does it modify?
Need a hint?
It changes the architecture — the family of functions the model can express.