Sometimes it's hard to put a score on something your eyes can see clearly.

Take a generated landing page. You can check for overlapping text, copy in the wrong tone, or a design that ignores the brand guidelines. Some of those checks can be deterministic; others need an LLM judge and examples of what counts as a failure.

But suppose two pages pass every check. On one, you can tell what the product does, why you'd want it, and where to go next. On the other, all the information is there, but you have to work to piece it together. Both meet the brief. I'd still pick the first page.

What score should it get? A 6? Or a 7?

That's the part of LLM judging I want to change. Keep the checks for known failures. For the harder question of which output is good, give the judge a definition and two options. Those head-to-head choices can become ratings over time, without asking the judge to invent a number for each output.

What Scores a 6 vs. a 7?

“Good” still needs a definition. For this page, I'd want a first-time visitor to understand the offer, find evidence supporting it, and know what to do next. If a visual treatment makes the explanation harder to follow, the explanation takes priority.

A typical pointwise judge would use that definition like this:

The judge has to assess the page and then translate that assessment onto a scale. It might correctly explain that the offer is clear but the supporting evidence is buried. I can agree with that explanation and still have no idea why it returned a 7. Giving examples of what each score means helps, but now those score boundaries need to hold up across very different pages.

I'd rather ask:

The judge can point to the difference: A puts the explanation next to the product screenshot; B makes you scroll past three sections of generic copy to find it. There's something here to agree or disagree with. Neither page needs an individual score, and a tie is fine when the distinction doesn't matter.

If you already have a pointwise judge that agrees reliably with people, keep it. The appeal here is how little extra interpretation the comparison requires.

Checking the Judge Against People

This is also how I'd check whether the judge is any good. Give a person the brief and the same two pages, hide which system produced each one, and ask them to choose. Collect the human and model decisions independently, with randomized presentation order.

If the person picks A and the judge picks B, pull up the pair. Did the judge reward decoration despite the rubric? Did the reviewer notice something the judge missed? Or did the brief leave a priority unclear? You can investigate without first deciding whether one reviewer's 6 means the same thing as another's 7.

Use a development set to work through those disagreements and revise the rubric. Test the revised judge on human-labeled pairs from held-out tasks, not just different pairs from tasks used to tune the rubric. Count agreement on A, B, and tie, break it down by task category, and have multiple people review some of the same pairs. Their agreement gives you a baseline for interpreting the judge's.

Make sure that test set includes pairs where both outputs pass the known-failure checks, and report agreement on those separately. A judge can agree with people about broken layouts and still be bad at choosing between two plausible designs. The MT-Bench and Chatbot Arena judge study is worth reading for its treatment of human agreement and judge biases.

Ranking the Configurations

With a judge you trust, you can start comparing system versions. Each configuration gets a rating: production, a new model, a revised prompt, a different retrieval pipeline. A match compares two configurations' outputs on the same task. The output itself doesn't keep a rating; the configuration that produced it does.

It's the same basic setup as Chatbot Arena, using an LLM judge in place of the human voter for offline evaluation.

I'd use online Elo for feedback as judgments arrive, then periodically fit Bradley–Terry to the comparison history for the reported ranking. Both turn pairwise results into relative ratings. The distinction matters because online Elo depends on match order; the batch fit doesn't.

You Don’t Need Every Matchup

Five configurations give you 10 possible pairs. Compare every pair on every one of 100 tasks and you've made 1,000 judge calls, before repeats or order-swapped checks.

You don't need to fill in that grid. Each judgment compares two outputs for the same task, but the rating model combines evidence across tasks and opponents. A connected set of comparisons can support predictions for pairs that haven't met.

After an initial batch, use the provisional ratings to choose useful follow-ups. If a revised prompt looks close to production, compare those two on more cases. Another round against the weakest configuration probably won't help you decide. Elo gives you the ratings; you still choose the matches.

When a sixth configuration arrives, compare it with established opponents without rerunning the old matchups. Their comparison history still counts, and cached outputs from unchanged versions can be reused. That assumes the judge, rubric, and task distribution remain comparable.

Close or noisy results still need more evidence. The saving comes from targeting unresolved comparisons, not a guarantee that a particular number of calls will be enough.

Try selecting a pair below and giving the win to the favorite, then to the underdog. The solid gray links stand for earlier comparisons across tasks; the dashed link is a matchup with no direct history.

Here, expected score means the predicted average result, counting one point for a win, half for a tie, and zero for a loss.

The ratings here are illustrative. The prediction comes from the rating gap, not a measured win rate between the selected pair, and doesn't show uncertainty.

Updating the Ratings

Elo moves the ratings according to how surprising a result was. For configurations rated $R_A$ and $R_B$, A's expected score is:

A win counts as 1, a tie as 0.5, and a loss as 0. Call that observed score $S_A$. After the judgment:

$K$ controls how much one result can move the ratings; B gets the opposite change. With a 200-point lead, A has an expected score of about 0.76. Set $K=16$ and a win adds only 3.8 points to A. A loss costs it 12.2. A tie narrows the gap too, since half a point is less than the model expected A to earn.

One detail worth keeping straight: with ties, expected score includes half the probability of a tie. It's only a win probability when there are no ties.

The starting value of 1,500 is arbitrary, and 400 sets the scale of the differences. A rating of 1,700 means something relative to the other configurations in this pool. It isn't an absolute quality score.

Which Pair to Compare Next

Check connectivity first. Groups with no comparisons linking them, directly or through shared opponents, can't be ranked relative to each other from the results. An unchanged production version makes a useful common opponent.

If one configuration mostly gets short product pages and another gets complicated pricing pages, you're mixing task differences into the ranking. Randomize pair assignments within each task category and track counts per configuration.

Reserve some budget for configurations with little history and under-tested task categories. A scheduler that only picks close ratings can keep overlooking both.

Adding a New Version

Suppose production and four alternatives already have ratings. You've changed the prompt and want to add v14 as a sixth configuration.

I'd first compare it with production and another established configuration on cases drawn from each task category. Generate or reuse the outputs from those exact versions, hide their identities, and ask the judge to choose. Only the two selected configurations need outputs for that comparison.

Keep enough in the comparison log to reproduce the decision: task ID and category, configuration versions, both outputs, failure-check results, judge and rubric versions, presentation order, and verdict. The rationale is useful when investigating a disagreement. Elo only uses the outcome.

Before shipping v14, I'd want uncertainty estimates around its difference from production and results for the important task categories, not just its place on the ladder.

Use the log for the Bradley–Terry refit, specifying how you handle ties. When estimating uncertainty, account for repeated judgments on the same task; they aren't independent new tasks. Arena's move from online Elo to Bradley–Terry explains the reporting approach.

If the judge, rubric, or task mix changes, re-run shared configurations on overlapping cases and check comparability before pooling old and new judgments. Otherwise, an unchanged configuration can appear to improve just because the evaluation changed.

What the Rating Leaves Out

Pairwise judges still have position bias and can favor longer outputs; length-controlled AlpacaEval addresses the latter problem. Check some pairs with the order swapped and keep sampling judgments for human review. Neither problem goes away because you put Elo after the judge.

A single rating can also hide useful differences. One configuration might produce better product pages and worse pricing pages. If preferences repeatedly cycle—A beats B, B beats C, C beats A—one rating per configuration may be a poor description of the results.

And keep the failure rates visible. If you only compare pages that pass the checks, a configuration's broken outputs never enter its ranking. Report its pass rate alongside that conditional rating. You still need to know how often it produces something usable.

Finally, a preference for a landing page isn't evidence that it converts better. That needs user data. This is an offline way to compare outputs while you're iterating.

Where I'd Start

I'd start with production, one proposed change, and a set of representative tasks. Write down what would make one output preferable, get human choices on a sample, and see where the judge disagrees. Keep the existing failure checks running alongside it.

Once those comparisons are useful, add configurations and build the ranking. The part I'd spend time on first is whether the judge picks the page I'd actually want to use. The rating update is a few lines of code.