Journal.

The authors test whether the internal representations of language models contain signals corresponding to the 26 CATO factors - the typical circumstances of trade secret disputes. On 7,893 short synthetic episodes in Qwen3-1.7B, Qwen3-8B and Gemma-2 2B, a linear classifier over activations recognizes each factor with AUC 0.998-1.000, and a single sparse autoencoder feature is enough for AUC from 0.86 to 1.00. For each factor in each model the layer and feature number are given, 78 coordinates in total. A simple word-based classifier is almost as accurate, and the method has not yet been tested on the texts of real decisions.

The author tests what language models do when the correct court holding is not among the offered options. In the CaseHOLD test of 5,314 questions, the correct holding was replaced with a similar one from another case in 1,670 of them, and a "none of the above" option was added. Six generative LLMs still pick a holding in 38-86% of such questions; the most cautious of four System One models, which return a probability for each option, does so in 30%. Accepting the best generative model's answer only when a System One model agrees halves the number of wrong citations. The author proposes evaluating legal AI services by the number of wrong citations, not by accuracy alone.

The authors build a neuro-symbolic model of Article 42 of the Dutch Bankruptcy Act, under which the trustee challenges the debtor's transactions that prejudice creditors. The structure of the rule is written by hand in the Catala language, while the evaluative concepts - the voluntary nature of the transaction, prejudice to creditors, the parties' knowledge - are decided by LLMs with retrieval over 846 Dutch court decisions. Splitting decisions into sections almost matched manual splitting (precision 0.98, recall 0.97 on 100 decisions). On ten difficult cases the model determined the outcome correctly in 6-7, it did worst on the parties' knowledge, and the number of retrieved decisions had no noticeable effect on the final figures.

The authors propose StrucChunk - a way to split statutes and contracts into fragments for retrieval in RAG systems along section boundaries rather than by token count. Each fragment is prefixed with its place in the document structure (act, chapter, section) and a summary of the sections it refers to; retrieval combines BM25 and BGE-M3 embeddings. On five datasets, from Indian procedural codes to the GDPR and Thai financial law, retrieval recall rises: on LegalBench-RAG Recall@8 is 48.18% versus 23.59%, on the Indian Code of Criminal Procedure Recall@5 is 0.804 versus 0.557. The main contribution comes from aligning fragment boundaries with section boundaries. The quality of the model's final answers was not measured.

The authors assembled LePhilBench, 489 US Supreme Court cases, and tested whether language models can identify a judge's legal views and use them for prediction. The test part has 220 opinions in 110 cases, labeled by three legally trained annotators along 14 dimensions of legal position, 9,240 labels in total. Ten models build a judge's profile from their opinions about as consistently with humans as humans with each other. Given a profile without the judge's name, the models predict the judge's answer to the key question more accurately: by 11.74 percentage points on average, by 31.42 on dissents. For court analytics this is a way to describe a judge by an explicit profile of legal views rather than by name.

A reward model compares two answers of a language model and picks the better one; the model is then fine-tuned on its signal. The authors assembled a benchmark of such evaluators for law: 1,331 pairs of answers to queries on Chinese law, 41 task types at five levels - knowledge of rules, text comprehension, analysis, decision and document drafting. The answers were generated by about 30 models ranging from 3 billion to 1 trillion parameters. Lawyers wrote rubrics based on reference answers, automatic scoring selected pairs with a noticeable gap in scores, and then five law graduate students checked each pair: 560 person-hours, with annotator agreement around 80%.

First place goes to Claude-Opus-4.7: 86.9% correctly chosen answers. Specially trained reward models scored from 62 to 78%, the legal model LegalOne-8B 74.7%, on par with general-purpose models of its size. The evaluators do worst at understanding long texts and at document drafting. A reference answer in the input raises accuracy by 3-8 points, while rubrics the model writes for itself give no consistent gain. The authors’ own reward model, trained on 39 thousand pairs, added 3.3 points to Qwen3-8B; adding a reference to the same model without training added 6.6.

The rankings matched practical usefulness: when selecting the best of four answers on LawBench and LexEval, the order of ten evaluators almost repeated the benchmark (Spearman correlation 0.98). For automatic checking of legal answers it follows that a reference and lawyer-verified criteria give more than fine-tuning a separate evaluator. The data is limited to Chinese law; the authors did not test transfer to other legal systems.

The paper When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection presents the PARCEL benchmark for checking whether a legal claim is supported by the source it cites. The authors collected 3,396 claims in parenthetical format based on recent decisions of the New York Court of Appeals, labeled “supported”, “contradicted” and “not found”, and tested current language models in zero-shot mode as a three-class inference task. The best models reach an accuracy of 0.97, yet they label unsupported claims as supported even when given the full text of the decision. The models detect the absence of support worse than a direct contradiction, and accuracy drops most on fabricated but plausible citations. The benchmark provides a way to measure the groundedness of answers from legal RAG systems at the level of an individual claim.

The paper studies how agents adapt to long legal proceedings at test time, without updating the model’s parameters. Two mechanisms are proposed: memory evolution, which extracts experience from previous cases, adjusts it to the factual and procedural context of the current case and accumulates it for subsequent decisions, and rubric alignment, which checks and corrects the actions of each role against behavioral and procedural requirements, linking roles and stages of the proceedings. Experiments on the J1-EVAL and LegalWorld datasets with five base models showed gains over reasoning approaches and agentic baselines at moderate interaction and computation costs. Ablations show that the two mechanisms complement each other: one is responsible for transferring experience, the other for coordinating roles.

In the paper Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness the authors study how far the judicial decisions of large language models depend on the legal rules they cite. The study was conducted by substituting the legal sources while keeping the facts of the case unchanged and analyzing the hidden states of seven models of different scales. It was found that the models often name the correct rules, yet the final verdicts do not change when those rules are replaced in a significant share of cases. This means that the presence of a reference to the law does not guarantee a logical link between the rule and the decision, so the generated reasoning cannot be used as an audit trail without a separate check of whether the verdict depends on the rule.

On July 31, Judge Engelmayer let Reddit's DMCA claims against SerpApi and Perplexity for circumventing SearchGuard and CAPTCHA proceed. We look at why disputes over scraping for AI are shifting from the CFAA to the anti-circumvention ban, and what the buyer of the data risks, not only the scraper.

On August 4, the Ninth Circuit vacated the injunction against the Comet AI agent working with Amazon accounts. We look at how the court resolved the question of access under the CFAA, why the agent's architecture mattered more than the label "autonomous AI", and what this means for those building agents.

The paper BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law tests models on the German method of subsumption: a rule is broken down into its elements, the facts are brought under each element in turn, and then a conclusion is formulated. The dataset contains 596 exam cases from different stages of legal education and 531 doctrinal problems; 12 systems were compared, including closed flagship models and open models. Evaluation by a judge model matched a human panel with a correlation of 0.96 on a sample of 30 papers. The authors note the small scale of human verification and the specifics of German legal education. For Lexometrica this is the closest analogue of the Rule and Application protocol in civil law: check each element of the rule and the explicit match of facts to elements, not only the final conclusion.

The paper JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment compares two ways of expert evaluation of legal answers: scores on a rubric and pairwise choice of the better of two. The dataset includes 30 real tasks, 1,539 rubric scores and 1,530 pairwise comparisons from practicing US attorneys; the same experts applied both protocols to the same materials. Pairwise comparisons recovered the given quality order with a Spearman correlation of 0.908, rubrics gave 0.150, and pairwise evaluation took less than half the time. A limitation: the answers were at three predefined quality levels, so the result cannot be transferred to closely matched models. For the Lexometrica leaderboard this is an argument to supplement absolute IRAC scores with blind pairwise comparisons on contested criteria.

The paper Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks tests models on tasks from Brazilian competitive examinations for judicial office. The dataset includes 74 tasks from eight exams held in 2023-2025, including practical exercises that require drafting a complete civil or criminal judgment: identify the issues, apply the rules to the facts, resolve the claims and draft the operative part. 23 models were evaluated, with grading performed by four independent frontier models; the dataset, the models’ answers and the evaluation code are published. A limitation: the primary grader remains a model, so biases shared across models are possible. For Lexometrica this is the format closest to predictive court analytics: tasks with a full judgment and separate evaluation of the reasoning and the final operative part.

Within seven weeks, three federal courts gave different answers on whether conversations with AI are protected from disclosure in litigation. We compare Heppner, Warner and Morgan: where the line runs between privilege and work product, what the party's status decides, and which contractual terms the court required from the AI platform.

Over the past year, federal appellate courts and state courts issued a series of sanctions for fabricated citations from AI tools: from $1,485 to disqualification from a case. We tabulate eight rulings and show which class of errors a metadata check misses, and why this is a question of model evaluation.

The paper Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queries tests whether the hallucination rate depends on place. The authors took scenarios from inquiries in family, housing, employment and traffic offence law, localized them for Los Angeles, London and Sydney, and gave the models’ answers to lawyers for labeling. The average hallucination rate was 0.45 for Los Angeles, 0.55 for London and 0.61 for Sydney; by area it was highest in housing law, 0.649, and lowest in employment law, 0.440. Consistency of answers under repeated generation correlated negatively with the hallucination rate. The authors do not establish the causes of the differences. For Lexometrica this is an argument to publish metrics by jurisdiction and area of law separately: an overall score hides the gap between well-represented and poorly represented rules.

The paper Are LLMs Court-Ready? Evaluating Frontier Models on Indian Legal Reasoning builds an Indian dataset of 6,218 multiple-choice questions from the CLAT exams, the Delhi judicial service and the Supreme Court of India’s Advocate-on-Record exam, plus 35 long-form tasks blind-graded by five certified Advocates-on-Record. Gemini 2.5 Pro exceeded the average results of the top candidates on the multiple-choice tests: by 7.6 points on CLAT UG and by 25.3 on DHJS. On the long-form tasks the model remained below the human results for each of the five graders; errors were noted in citation discipline, procedural requirements and document structure. For Lexometrica this confirms that multiple-choice tests and the accuracy of the conclusion are not enough: separate assessments of procedure, relevance of sources and IRAC structure are needed.

The paper LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation proposes evaluating a legal answer not as a whole but by atomic claims, Legal Data Points. Each receives a label of correct, incorrect, irrelevant or missing, and from the labels correctness, precision, recall and F1 are computed. On a subset of LegalBench the Pearson correlation with expert assessment of correctness was 0.700 versus 0.018 for DeepEval Correctness. Inter-expert agreement when labeling in the LeMAJ interface rose from 0.77 to 0.88. The authors acknowledge the subjectivity of the relevance criterion and the dependence on the judge model. For the Lexometrica leaderboard, separate accounting of missing mandatory elements of an answer is useful: a model can be right in everything it says and still fail to name the rule without which the conclusion is unfounded.

The paper L-MARS: Legal Multi-Agent System with Agentic Search and Citation-Faithfulness Audit distributes a legal question among agents for planning, search, source filtering, verification and answer synthesis. For evaluation, LegalSearchQA was created: 50 questions across five areas of law whose answers depend on events after the models’ training cutoff. The system reached 96% accuracy versus 58% without search, while an ordinary chain of thought gave 30%: extended internal reasoning without reliable search made the result worse. The dataset is small, and the answers require updating as the law changes. Two conclusions matter for Lexometrica: compare agentic search not only with a plain query but also with chain of thought and single-pass RAG, and version the currency date of each case and of the legislation corpus.