JudgmentBench: rubrics versus pairwise judgments

The paper JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment compares two ways of expert evaluation of legal answers: scores on a rubric and pairwise choice of the better of two. The dataset includes 30 real tasks, 1,539 rubric scores and 1,530 pairwise comparisons from practicing US attorneys; the same experts applied both protocols to the same materials. Pairwise comparisons recovered the given quality order with a Spearman correlation of 0.908, rubrics gave 0.150, and pairwise evaluation took less than half the time. A limitation: the answers were at three predefined quality levels, so the result cannot be transferred to closely matched models. For the Lexometrica leaderboard this is an argument to supplement absolute IRAC scores with blind pairwise comparisons on contested criteria.