Can LLMs abstain from a wrong citation

US courts sanction lawyers for citing decisions that do not exist and decisions that do not support the stated proposition: a public tracker counts 2,203 such cases, 1,264 of them in 2026, and in 933 a real decision is cited for a proposition it does not support. The author tests whether models can decline to answer when no correct source exists. The basis is the CaseHOLD set: an excerpt from a US court decision with the brief statement of the cited case’s holding removed, and five candidate holdings, one correct and four most similar ones from other citations. In the abstention condition a sixth option, “none of the holdings fits”, is added, and in 1,670 of 5,314 test questions (31%) the correct holding is replaced with a similar holding from another case, so the sixth option becomes the right answer.

Four System One models are compared - Jev, pplx-decider, d1-ultra and GPT-6 Luna Decisions, which are built on LLMs but return a probability for each option instead of text - against six generative LLMs from five vendors, from Gemini 3.8 Flash to GPT-4o-mini, with the same task and temperature 0. Errors are counted separately: choosing a holding when none is correct, and abstaining when a correct one exists. Three ways of combining the two model types are tested, including accepting an answer only when both agree; thresholds are tuned on 30% of the questions and results are reported on the remaining 3,720. Finally, a local Qwen3-8B is fine-tuned on 10,000 examples using only Jev’s probabilities, without labels.

On the original test, pplx-decider and Gemini 3.8 Flash match in accuracy (88.1% and 88.2%), but the former costs roughly 55 times less. When no correct holding exists, generative models still pick one of the offered holdings in 37.8-85.6% of cases, System One models in 30.1-60.8%. The most cautious one, Jev, abstains in 26.2% of questions where a correct answer existed; generative models have fewer than 8% such abstentions. Gemini 3.8 Flash on its own gives a wrong citation in 18.4% of questions; if its answer is accepted only when pplx-decider agrees, the share drops to 9.4%, while 12.7% of questions go to manual review instead of 3.0%. Qwen3-8B trained on Jev’s probabilities reproduces its accuracy (63.8% versus 63.9%) and is better calibrated than its teacher: a calibration error of 0.025.

The author names the limitations himself. The absence of a holding is modeled in one way, by substituting another case’s holding, whereas in practice a decision may have been overturned, belong to another jurisdiction, or not exist at all. CaseHOLD was assembled from decisions published before 2021, and the models may have seen them in training. How much worse a wrong citation is than an unnecessary review is unknown, so the ratios of 1, 5 and 10 in the calculations are arbitrary. Only US case law in English was tested, with one prompt per model type. For court analytics and legal services the practical conclusion of the paper is this: alongside accuracy, report the share of citations to an unsuitable decision and the share of unnecessary abstentions, because these errors cost differently, and the choice between them, in the author’s view, belongs to the lawyer who uses the system.