intermediate

Model Evaluation & Validation

Metrics that matter, validation strategies that don't leak, and training techniques that generalize.

Precision & RecallF1 ScoreROC-AUCConcordance & Somers' DDecile & Gains TablesKS Statistic & LiftDeLong & McNemar TestsPR CurvesConfusion Matrixk-Fold Cross-ValidationData LeakageBootstrappingEarly StoppingHyperparameter SearchLearning Rate SchedulesData AugmentationPerplexity & Bits-per-ByteLLM BenchmarksAI-as-a-JudgePairwise Eval & EloModel Selection WorkflowBuild-vs-BuyCost & LatencyEmpirical Risk MinimizationGeneralization & VC DimensionDouble DescentCalibration & ECEAleatoric vs Epistemic Uncertainty
ML project loop2/6 stages →
DataHypothesis spaceObjectiveOptimizationEvaluationDeployment feedback

9 lessons

01

Classification & Regression Metrics

Precision, recall, F1, ROC-AUC, PR curves — choosing the right metric for the right problem.

15m
02

Validation Strategies

Train/test splits, k-fold CV, data leakage, and bootstrapping — getting reliable generalization estimates.

14m
03

Training Techniques & Hyperparameter Search

Early stopping, learning rate schedules, grid/random/Bayesian search, and data augmentation.

16m
04

LLM Evaluation

Information-theoretic intrinsic metrics (cross-entropy, perplexity, bits-per-byte), capability benchmarks (MMLU, HellaSwag, GSM8K, HumanEval, BIG-bench), and the modern preference-based stack — AI-as-a-judge with its biases, pairwise comparison, and Bradley–Terry / Elo ratings.

22m
05

Evaluating AI Systems

The systems-level companion to LLM evaluation: how to pick a production model — task-specific eval sets, build-vs-buy economics, the limits of public benchmarks, and treating cost / latency / calibration as first-class metrics.

20m
06

Learning Theory & Generalization

Why models generalize — empirical risk minimization, the bias-variance decomposition, the generalization gap, VC dimension and capacity, and the surprise of double descent.

24m
07

Calibration & Uncertainty

When a model says 90%, is it right 90% of the time? Reliability diagrams and ECE, temperature scaling, and the difference between aleatoric and epistemic uncertainty.

18m
08

Rank Ordering & Model Comparison

Concordance, decile and gains tables, KS and lift, and the paired statistical tests (DeLong, McNemar, the bootstrap) that decide whether one model is actually better than another.

18m
09

Model Evaluation Quiz

Test your understanding of metrics, validation strategies, training techniques, rank ordering, and learning theory.

12m