Model Evaluation & Validation
Metrics that matter, validation strategies that don't leak, and training techniques that generalize.
9 lessons
01Classification & Regression Metrics
Precision, recall, F1, ROC-AUC, PR curves — choosing the right metric for the right problem.
Validation Strategies
Train/test splits, k-fold CV, data leakage, and bootstrapping — getting reliable generalization estimates.
Training Techniques & Hyperparameter Search
Early stopping, learning rate schedules, grid/random/Bayesian search, and data augmentation.
LLM Evaluation
Information-theoretic intrinsic metrics (cross-entropy, perplexity, bits-per-byte), capability benchmarks (MMLU, HellaSwag, GSM8K, HumanEval, BIG-bench), and the modern preference-based stack — AI-as-a-judge with its biases, pairwise comparison, and Bradley–Terry / Elo ratings.
Evaluating AI Systems
The systems-level companion to LLM evaluation: how to pick a production model — task-specific eval sets, build-vs-buy economics, the limits of public benchmarks, and treating cost / latency / calibration as first-class metrics.
Learning Theory & Generalization
Why models generalize — empirical risk minimization, the bias-variance decomposition, the generalization gap, VC dimension and capacity, and the surprise of double descent.
Calibration & Uncertainty
When a model says 90%, is it right 90% of the time? Reliability diagrams and ECE, temperature scaling, and the difference between aleatoric and epistemic uncertainty.
Rank Ordering & Model Comparison
Concordance, decile and gains tables, KS and lift, and the paired statistical tests (DeLong, McNemar, the bootstrap) that decide whether one model is actually better than another.
Model Evaluation Quiz
Test your understanding of metrics, validation strategies, training techniques, rank ordering, and learning theory.