BenGER: a benchmark of subsumption in German law

The paper BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law tests models on the German method of subsumption: a rule is broken down into its elements, the facts are brought under each element in turn, and then a conclusion is formulated. The dataset contains 596 exam cases from different stages of legal education and 531 doctrinal problems; 12 systems were compared, including closed flagship models and open models. Evaluation by a judge model matched a human panel with a correlation of 0.96 on a sample of 30 papers. The authors note the small scale of human verification and the specifics of German legal education. For Lexometrica this is the closest analogue of the Rule and Application protocol in civil law: check each element of the rule and the explicit match of facts to elements, not only the final conclusion.