A precedent retrieval benchmark for legal analysis

The paper A Reasoning-Focused Legal Retrieval Benchmark, by Lucia Zheng, Neel Guha and co-authors from Stanford, builds a retrieval benchmark over US case law. Unlike datasets where a document must be found by similarity to the query, the tasks require identifying the sources needed for legal analysis and then producing a reasoned answer. The main methodological contribution: retrieval and generation of the analysis are evaluated separately, so it is visible whether the system failed to find the controlling precedent, picked the wrong case, or misapplied what it found. Limitations: US law, the English language and the structure of American citations. For Lexometrica the approach transfers to Russian cases: evaluate separately the retrieval of the applicable rule, the retrieval of practice, and the consistency of the conclusion with the sources found.