Mediocrity is the key for LLM as a Judge Anchor Selection
The ``LLM-as-a-judge'' paradigm has become a standard method for evaluating open-ended generation. To address the quadratic scalability costs of pairwise comparisons, popular benchmarks like Arena-Hard and AlpacaEval compare all models against a single anchor. However, despite its widespread use, the impact of anchor selection on the reliability of the results remains largely unexplored. In this work, we systematically investigate the effect of anchor selection by evaluating 22 different anchors on the Arena-Hard-v2.0 dataset. We find that the choice of anchor is critical: a poor anchor can dramatically reduce correlation with human rankings. We identify that common anchor choices (best-p...
| Authors | Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen et al. |
|---|---|
| Primary category | cs.CL |
| Published | 17 March 2026 |
| Last updated at source | 2 September 2026 |
Models and datasets referencing this paper
- ibm-research/900K-JudgementsDataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.