LeMAJ: atomic evaluation of legal answers

The paper LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation proposes evaluating a legal answer not as a whole but by atomic claims, Legal Data Points. Each receives a label of correct, incorrect, irrelevant or missing, and from the labels correctness, precision, recall and F1 are computed. On a subset of LegalBench the Pearson correlation with expert assessment of correctness was 0.700 versus 0.018 for DeepEval Correctness. Inter-expert agreement when labeling in the LeMAJ interface rose from 0.77 to 0.88. The authors acknowledge the subjectivity of the relevance criterion and the dependence on the judge model. For the Lexometrica leaderboard, separate accounting of missing mandatory elements of an answer is useful: a model can be right in everything it says and still fail to name the rule without which the conclusion is unfounded.