ML in Practice Quiz

ML in Practice Quiz

Check your understanding across the course — feature leakage and drift, scaling and calibration, rollout patterns (shadow / canary / A/B), PSI thresholds, reproducibility, KV-cache memory, and choosing a fairness definition.

Exercise

A data scientist fits a StandardScaler on the ENTIRE dataset (train + test) before splitting into train/test. What kind of problem does this introduce?

Need a hint?

What information from the test set is now visible during training?

Exercise

A model monitors feature distributions and detects that the 'age' feature has shifted from mean=35 to mean=42 over 6 months, while model accuracy on labeled samples remains the same. What type of shift is this, and should you retrain?

Need a hint?

The inputs changed but the accuracy didn't — what does that tell you about P(Y|X)?

Exercise

Which of these algorithms REQUIRES feature scaling for correct results, not just for faster convergence?

Need a hint?

Think about which algorithm's core computation is directly affected by feature magnitude (not just convergence speed).

Exercise

A SHAP summary plot shows that 'income' has both large positive and large negative SHAP values depending on the sample. What does this tell you about how the model uses this feature?

Need a hint?

Positive SHAP = pushes toward positive class; negative = pushes toward negative class. Both extremes for the same feature means...?

Exercise

A weather forecast model predicts 90% chance of rain on 100 days. It rains on 90 of those days. A loan default model predicts 90% default probability for 100 loans; 90 default. Which model is better calibrated?

Need a hint?

Calibration compares predicted probability to observed frequency across many predictions.

Exercise

Your team finishes 'shadow' on a new ranker — offline cross-entropy on live traffic looks great. The PM wants to jump straight to a 50/50 A/B test for faster statistical power. What is the strongest argument for inserting a small-fraction canary (1–5%) BEFORE the A/B?

Need a hint?

Shadow throws away the candidate's outputs. A/B exposes 50% of users. What gap sits between those two patterns?

Exercise

Your monitoring dashboard reports PSI = 0.18 on the 'device_type' feature for three consecutive daily windows. The standard PSI playbook treats <0.1 as stable, 0.1–0.25 as watch, and >=0.25 as act. What is the right response?

Need a hint?

Where does 0.18 fall on the standard threshold table — and how does sustained vs transient change what you do?

Exercise

Six weeks after training, you need to reproduce a benchmark run bit-for-bit. Your tracker logged the git SHA of the training repo and the final metrics. What is the MINIMUM additional pair of artifacts you need to make the run truly reproducible?

Need a hint?

The lesson names three things you must version. Code is one — what are the other two?

Exercise

You serve a decoder-only LLM at batch size 16 with 4k context. To free more GPU memory for concurrent requests, your team is choosing between (i) shrinking the model from FP16 to INT4 weights and (ii) quantizing the KV-cache to FP8. The serving job is currently OOM-ing on the KV-cache, not the weights. Which lever wins, and why?

Need a hint?

Recall: KV-cache bytes scale as 2 * L * H * d * s * b * bytes. Which lever attacks that term directly?

Exercise

You build a loan-approval classifier. Stakeholders agree the harm to mitigate is qualified applicants being denied at different rates across demographic groups. Base rates of repayment differ across groups in the historical data. Which fairness definition is the BEST fit, and why can you not also enforce demographic parity?

Need a hint?

Which definition conditions on Y=1? And what does the Chouldechova/Kleinberg impossibility result say when base rates differ?

Exercise

A feature store builds a training row for a purchase that happened in March using the customer's CURRENT lifetime-spend value. What's wrong?

Need a hint?

What did the model 'know' at prediction time that it couldn't have known in March?

Exercise

In differential privacy, what does a smaller privacy budget ε guarantee?

Need a hint?

Pr[M(D) ∈ S] ≤ e^ε · Pr[M(D′) ∈ S] for datasets differing in one person.

Exercise

A deployed model's accuracy decays over months as the world shifts, and a monitor triggers a retrain on fresh data. Which slot of the ML project loop is this cycle?

Need a hint?

It happens after deployment and closes the loop back to the start.

Exercise

Transfer test — a technique this course did NOT teach: the kernel trick, which maps data into a higher-dimensional space so a linear model can fit curved boundaries. Which slot does it modify?

Need a hint?

It enlarges the set of functions the model can represent.