LegalRewardBench: evaluating reward models in law

A reward model compares two answers of a language model and picks the better one; the model is then fine-tuned on its signal. The authors assembled a benchmark of such evaluators for law: 1,331 pairs of answers to queries on Chinese law, 41 task types at five levels - knowledge of rules, text comprehension, analysis, decision and document drafting. The answers were generated by about 30 models ranging from 3 billion to 1 trillion parameters. Lawyers wrote rubrics based on reference answers, automatic scoring selected pairs with a noticeable gap in scores, and then five law graduate students checked each pair: 560 person-hours, with annotator agreement around 80%.

First place goes to Claude-Opus-4.7: 86.9% correctly chosen answers. Specially trained reward models scored from 62 to 78%, the legal model LegalOne-8B 74.7%, on par with general-purpose models of its size. The evaluators do worst at understanding long texts and at document drafting. A reference answer in the input raises accuracy by 3-8 points, while rubrics the model writes for itself give no consistent gain. The authors’ own reward model, trained on 39 thousand pairs, added 3.3 points to Qwen3-8B; adding a reference to the same model without training added 6.6.

The rankings matched practical usefulness: when selecting the best of four answers on LawBench and LexEval, the order of ten evaluators almost repeated the benchmark (Spearman correlation 0.98). For automatic checking of legal answers it follows that a reference and lawyer-verified criteria give more than fine-tuning a separate evaluator. The data is limited to Chinese law; the authors did not test transfer to other legal systems.