In the paper LegalReasoner: Step-wised Verification-Correction for Legal Judgment Reasoning, predicting the outcome of a case is built as a verifiable process rather than classification of the result. The model identifies the disputed issues, builds a chain of reasoning and scores each step for correctness, progress and potential. For training, the LegalHK dataset of 58,130 Hong Kong cases was assembled. Accuracy of the conclusion on LegalHK was 80.27 versus 72.37 for a fine-tuned Llama-3.1-70B and 77.15 for GPT-o1; on the Chinese CMDL it was 76.95 versus 72.03. Without step-by-step reasoning the result dropped to 69.49, without verification and correction of steps to 74.49. The authors point to the cost of verifying each step and the noise of automatic labeling. For the Lexometrica leaderboard this is an argument to score Issue, Rule, Application and Conclusion separately rather than with a single overall score.
A precedent retrieval benchmark for legal analysis
The paper A Reasoning-Focused Legal Retrieval Benchmark, by Lucia Zheng, Neel Guha and co-authors from Stanford, builds a retrieval benchmark over US case law. Unlike datasets where a document must be found by similarity to the query, the tasks require identifying the sources needed for legal analysis and then producing a reasoned answer. The main methodological contribution: retrieval and generation of the analysis are evaluated separately, so it is visible whether the system failed to find the controlling precedent, picked the wrong case, or misapplied what it found. Limitations: US law, the English language and the structure of American citations. For Lexometrica the approach transfers to Russian cases: evaluate separately the retrieval of the applicable rule, the retrieval of practice, and the consistency of the conclusion with the sources found.
Hallucination-Free?: legal RAG hallucination rates
The paper Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, published in the Journal of Empirical Legal Studies, tests three commercial legal research tools: Lexis+ AI, Westlaw AI-Assisted Research and Ask Practical Law AI. The study was preregistered; the authors wrote legal queries, verified claims and citations, and distinguished a hallucination, a factual error and a substantive answer. Despite RAG and access to databases, the systems hallucinated in 17-33% of answers. That is lower than general-purpose GPT-4 but does not confirm the vendors’ claims of reliability. A separate finding: a refusal or an evasive answer lowers the hallucination rate without making the product more useful. For the Lexometrica leaderboard this is grounds to treat citation accuracy, content accuracy and the share of answered queries as different metrics.
Thomson Reuters v. ROSS: headnotes and fair use
On February 11, Judge Bibas held that training ROSS's legal-search product on Westlaw headnotes infringes copyright and rejected fair use. We look at what exactly the court treated as copying, why the ruling did not become a general rule for LLMs, and what it means for the provenance of training data.