LLM Evaluation & LLM-as-a-JudgeOpen in Colab
Agentic project loop
Task definition·Context & tools·Orchestration loop·Evaluation·Guardrails·Operations feedback

LLM Evaluation & LLM-as-a-Judge

An LLM app has no test suite by default. The output is open-ended text, "correct" is fuzzy, the model is stochastic, and a one-line prompt tweak can silently regress a thousand answers. Evaluation is the discipline that turns "it seems better" into a number you can defend, and it is the single biggest difference between a demo and a product.

Real-world examples: deciding whether GPT-class model A or B is better for your summariser; catching that a "small" prompt edit dropped answer quality 8%; gating a RAG release on a faithfulness score before it reaches users.

The evaluation landscape

Pick the cheapest method that actually measures what you care about:

  • Reference-based: compare output to a gold answer. Cheap and deterministic when a reference exists: exact match / F1 for extraction, BLEU/ROUGE for translation/summarisation (surface overlap), embedding similarity for "close enough".
  • Reference-free: no gold answer; score a property of the output directly: schema validity, toxicity, faithfulness to retrieved context (does the answer follow from the sources?).
  • Human evaluation: the gold standard for nuanced quality, but slow and expensive; used to calibrate the cheaper automated methods rather than run on every release.

For free-running text generation, a base perplexity PPL=exp⁡ ⁣(−1N∑ilog⁡p(xi))\text{PPL} = \exp\!\big(-\tfrac{1}{N}\sum_i \log p(x_i)\big) measures fit to a distribution but says little about task quality — low perplexity text can still be unhelpful. Task metrics matter more than language-modelling metrics.

BLEU/ROUGE score n-gram overlap against a reference and predate LLM-as-a-judge by two decades — they're still the default for translation/summarisation, but their surface-overlap blind spots are exactly what motivates the model-based approach below.

Deep dive
Text Generation Metrics: BLEU, ROUGE & METEOR
Includes runnable notebook

LLM-as-a-judge

When outputs are open-ended and no reference exists, the scalable option is an LLM-as-a-judge: prompt a strong model to score or compare outputs against a rubric.

  • Pointwise: "rate this answer 1–5 on faithfulness." Convenient but poorly calibrated.
  • Pairwise: "which answer is better, A or B?" More reliable, because models rank better than they grade. This is how preference leaderboards (e.g. arena-style Elo) are built.

Judges are biased — design around it

LLM judges exhibit measurable biases: a position bias (favouring the first option), a verbosity bias (favouring longer answers), and a self-preference bias (favouring outputs from the same model family). Mitigate by swapping A/B order and averaging, controlling for length, and validating the judge against human labels before trusting it. An unvalidated judge is an opinion, not a metric.

Eval suites and regression testing

The unit that makes evaluation operational is the eval suite: a versioned dataset of representative inputs plus the metrics to run on them, executed automatically on every prompt, model, or retrieval change — exactly like a software test suite.

score=1∣D∣∑x∈Dmetric(system(x))\text{score} = \frac{1}{|D|}\sum_{x \in D} \text{metric}\big(\text{system}(x)\big)

Run it in two places:

  • Offline (pre-deploy): gate a release: block the change if the suite score drops or any safety check fails.
  • Online (post-deploy): sample real production traffic, score it continuously, and watch for drift in quality that the offline set didn't capture.

Curate the eval set from production failures

The most valuable eval examples are the ones your system got wrong in production. Build a feedback loop: capture failures and edge cases, add them to the suite, and your evaluation gets more discriminating over time, while a static benchmark slowly goes stale.

Exercise

Which is a documented bias of an LLM-as-a-judge that you must design around?

Need a hint?

Think about answer order and answer length.

Exercise

You're evaluating an open-ended chatbot with no gold reference answers. Which approach fits best?

Need a hint?

Surface-overlap metrics need a reference; this task has none.

Common mistakes

  1. Shipping on vibes: a few hand-checked examples instead of a versioned eval suite.
  2. Trusting an unvalidated judge: using LLM-as-a-judge scores as ground truth without checking agreement with humans.
  3. Reference metrics for open-ended tasks: reporting BLEU/ROUGE for a chatbot, where surface overlap barely correlates with quality.
  4. Pointwise where pairwise wins: asking for absolute 1–5 scores when "A vs B" would be far more reliable.

Key takeaways

  • Choose the cheapest valid method: reference-based when a gold answer exists, reference-free for properties like faithfulness, human to calibrate the rest.
  • LLM-as-a-judge scales open-ended evaluation; prefer pairwise and design around position/verbosity/self-preference bias.
  • An eval suite is a versioned dataset + metrics, run offline as a release gate and online on sampled traffic.
  • Grow the suite from real production failures so it sharpens instead of staling.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
LLM Evaluation & LLM-as-a-JudgeAttention & TransformersNatural Language Processi…Optimizing LLM InferenceChain-of-Thought & Struct…Retrieval-Augmented Gener…AI-Engineering Architectu…RAG Architectures: Which …Evaluating AgentsLLM-as-a-Judge: Automated…Text Generation Metrics: …The Agentic Project Loop