Evaluating LLM Applications: Benchmarks, Evals and LLM-as-Judge

The most common way an AI feature fails is not dramatically. It ships, it seems fine in a demo, and then it quietly gets worse — a prompt tweak here, a model upgrade there — until someone notices the summaries have been wrong for three weeks. Nothing errored. There was no test to fail.

Why public benchmarks do not help

Leaderboard scores measure general capability on academic tasks. Your application does one specific thing on your specific data, and the correlation between "scores well on a reasoning benchmark" and "extracts the right delivery date from our supplier emails" is weak.

Worse, benchmarks leak into training data over time, so a rising score can reflect contamination rather than capability. Use them to shortlist models. Never use them to decide whether your feature works.

Build an eval set from real traffic

The eval set is the deliverable that makes everything else possible, and it does not need to be large. Fifty to two hundred well-chosen examples beats ten thousand random ones.

1. Start from real inputs. Log what users actually send. Synthetic examples you invent will be cleaner and more polite than reality.
2. Include the failures. Every bug report, every thumbs-down, every case someone escalated. These are the highest-value rows in the set.
3. Cover the distribution, not just the average. Short inputs and very long ones, the common language and the rare one, well-formed requests and gibberish, and the adversarial cases someone will eventually try.
4. Write the expected outcome — either an exact answer, or a checklist of properties the answer must have.
5. Version it. The eval set is code. It lives in the repository and changes through review.

Choose the cheapest metric that works

Not everything needs a model to grade it. A ladder from cheapest to most expensive:

• Deterministic checks. Is it valid JSON? Does it match the schema? Is the extracted date parseable and within a plausible range? Does it cite a document that exists? A large share of real failures are caught here for essentially zero cost.
• String and structural comparison. Exact match for classification, set overlap for extraction, unit tests for generated code.
• Reference-based similarity. Embedding distance to a known-good answer. Useful as a smoke signal, unreliable as a verdict.
• Model-graded. Another model scores the output against a rubric. Necessary for open-ended text, and the most fragile of the four.
• Human review. The ground truth. Expensive, so spend it calibrating the automated graders rather than grading everything.

Most teams jump straight to model-graded evaluation and skip the first two, which is backwards — the deterministic checks are the ones that never disagree with themselves.

LLM-as-judge, done carefully

Using a model to grade output works, with caveats that matter:

You are grading a customer support reply.

TICKET: {ticket}
REPLY: {reply}
POLICY: {relevant_policy_extract}

Score each criterion 1-5 and give a one-sentence reason:
1. Factual accuracy — every claim is supported by POLICY.
2. Completeness — every question in TICKET is addressed.
3. Tone — professional, direct, no hedging.

Then output: {"accuracy": n, "completeness": n, "tone": n, "reasons": {...}}

What makes the difference between a useful judge and a random number generator:

• Specific criteria, not "is this good". A vague rubric produces scores clustered around 4 with no discrimination.
• A narrow scale. Models cannot meaningfully distinguish 7 from 8 on a ten-point scale. Use 1–5, or binary pass/fail per criterion, which is more reliable still.
• Reasons before scores. Asking for the justification first produces better-calibrated scores than asking for the number first.
• Give it the evidence. A judge grading factual accuracy without the source document is guessing.
• Watch for known biases. Judges favour longer answers, answers that resemble their own style, and — in pairwise comparison — whichever option was presented first. Randomise position and run both orders.
• Calibrate against humans. Have a person grade fifty examples, compare with the judge, and adjust the rubric until they broadly agree. Without this step you do not know whether your judge measures quality or verbosity.

Wire it into the pipeline

An eval suite that runs manually before a big change will not run before the small change that breaks things. Treat it like tests:

run:  every prompt change, model change, retrieval change, dependency bump
gate: block the merge if the pass rate drops more than N points
track: store per-case results so you can see which case regressed, not just the average

The per-case history matters more than the headline number. An aggregate that moves from 87% to 85% is ambiguous; knowing that three specific cases about refund policy started failing is actionable.

Evaluate the whole system, not just the model

For a retrieval-augmented feature, a wrong answer has at least three possible causes, and the fix is different for each:

• Retrieval missed the relevant document — a chunking or embedding problem.
• Retrieval found it and the model ignored it — a prompt or context-ordering problem.
• The document itself is wrong or outdated — a content problem, and the most common one in practice.

Measure each stage separately. Retrieval recall at k tells you whether the right chunk was even available. Without that, you will spend weeks tuning prompts to fix a search problem.

Production is the real eval

Offline evals catch regressions; they do not tell you whether users are getting value. Instrument the live feature with implicit signals — did the user accept the suggestion, edit it heavily, retry, or abandon the flow — and sample real outputs for periodic human review. When a new failure mode appears in production, it becomes a new row in the eval set, and the loop closes.