LLM-as-a-Judge: How to Evaluate AI Agents Without Fooling Yourself

You changed one line of a system prompt and the agent feels better. Is it? Without evaluation you are shipping on vibes. Human review is the gold standard but does not scale past a few dozen examples. That is why most teams now use LLM-as-a-judge: a model grading another model's output against a rubric. Done well, it is the backbone of agent quality. Done casually, it is an expensive way to agree with yourself.

What LLM-as-a-judge is (and where the idea came from)

The approach was studied systematically in the 2023 paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al.). The authors found that strong models' judgments agreed with human preferences at roughly the same rate humans agreed with each other, over 80% in their setup. They also documented the biases that still matter today: position bias, verbosity bias and self-enhancement bias.

There are three common formats:

• Pointwise scoring. Grade one answer on a rubric (for example 1 to 5 for faithfulness). Easy to track over time.
• Pairwise comparison. Which of two answers is better? More reliable for small differences, ideal for A/B testing prompts or models.
• Reference-based. Compare the answer to a known-good reference. Best when a correct answer exists.

Write rubrics a judge can actually follow

Vague criteria like "helpfulness" produce noisy scores. Break quality into narrow, observable checks and ask for a short justification before the score:

You are grading an answer from a support agent.

Criteria (answer each independently):
1. FAITHFUL: Every factual claim is supported by the provided sources. (yes/no)
2. COMPLETE: The answer addresses every part of the user's question. (yes/no)
3. POLICY: The answer does not promise refunds beyond 30 days. (yes/no)
4. TONE: Polite and concise, under 150 words. (1-3)

First write one sentence of reasoning per criterion, quoting the text you relied on.
Then output JSON: {"faithful":bool,"complete":bool,"policy":bool,"tone":1|2|3}

The biases you must design around

• Position bias. In pairwise mode, judges tend to prefer one slot. Run each pair twice with the order swapped and only count consistent wins.
• Verbosity bias. Longer answers look more thorough. Add explicit length limits to the rubric, or compare length-matched answers.
• Self-preference. A model may favour outputs written in its own style. Where possible, judge with a different model family than the one being tested.
• Leniency drift. Judges are often too generous. Include known-bad examples in your eval set to check that they actually fail.

Calibrate the judge against humans first

Before trusting a judge, have two people label 50 to 100 examples using the same rubric. Measure how often the judge agrees with the humans, and how often the humans agree with each other. If human-human agreement is low, your rubric is ambiguous, and no judge will fix it. Revise the rubric until people agree, then tune the judge prompt until it matches them. Re-check whenever you change the judge model.

Evaluating agents, not just answers

Agents fail in the process, not only the final message. A correct answer reached by calling the payments API six times is still a bug. Grade the trajectory too:

• Tool selection. Did it call the right tool for each step?
• Arguments. Were tool inputs valid and grounded in the conversation?
• Efficiency. Number of steps, tokens and cost compared with a reference run.
• Safety. Did it stay within allowed tools and ask for approval when required?
• Outcome. Did the final state match the goal (ticket closed, row created, correct total)?

Prefer deterministic checks wherever they exist: exact matches on tool names, schema validation on arguments, database assertions on the end state. Use the LLM judge only for what code cannot check, such as faithfulness and tone. Our post on Pydantic as the contract layer for agents shows how typed outputs make many of these checks trivial.

Run evals like tests

1. 1Keep a versioned eval set in your repo: inputs, expected outcomes, and tags such as refunds or edge-case.
2. 2Run it on every prompt, tool or model change, in CI or before deploy.
3. 3Fail the build when a key metric drops beyond a set threshold, not on any change at all, because scores are noisy.
4. 4Feed production failures back into the set every week. The best eval sets grow from real incidents.
5. 5Sample live traffic for ongoing judging so you notice drift between releases. See our guide to LLM observability.

Open-source tools worth knowing

You do not need a platform to start: a script, a JSON file of examples and a judge prompt get you surprisingly far. When you outgrow that, popular open-source options include promptfoo (config-driven evals and red-teaming), DeepEval (pytest-style LLM tests), Ragas (RAG-specific metrics) and the evaluation features in Langfuse and Arize Phoenix. Each has different strengths, so try them on your own eval set before standardising.

Practise it hands-on

The evaluation notebooks in the AgentSwarms notebooks library walk through golden sets, rubric-based judging and regression checks in runnable cells. When your agents graduate to production on the self-hosted platform, the same habits apply: version the eval set, judge with a different model, and track scores next to cost.