Predicting a ruling from a judge’s legal views

The authors test whether language models can work with a judge’s legal philosophy - their views on the nature and sources of law, because of which judges relying on the same rules and precedents reach different decisions. For this the LePhilBench dataset was assembled: 489 US Supreme Court cases from December 2005 to June 2025, each with one opinion of the court and one dissenting opinion giving opposite answers to the key question of the case. Together with a professor of constitutional law the authors drew up 14 dimensions of legal position, each a pair of opposing views, for example natural law versus legal positivism or conservative versus liberal views on the role of the state. The test part, 110 cases and 220 opinions, was labeled by three annotators with legal training on all 14 questions, 9,240 labels in total; agreement between them is moderate, Fleiss’ kappa 0.474.

There are two tasks. In the first, the model reads an opinion and answers the 14 questions itself; its answers are compared with the human labels. In the second, the model receives the facts of the case, the key question and the judge’s profile assembled from their past opinions, but not the judge’s name and not the side they took, and predicts the judge’s answer, yes or no. Since the opinion of the court and the dissent answer the question in opposite ways, accuracy without a profile is 50% by construction, and the gain over that level shows the contribution of the legal position. Ten models from the GPT-5.6, Claude 5, Qwen3.7, Gemini 3.7 and Muse Spark families were tested, each experiment repeated three times.

In the first task the models err where humans disagree: on the five dimensions with the highest annotator agreement (76.4-81.2%) the models also have the highest agreement with humans (75.3-78.9%), and on the two most contested ones the lowest. Most often the model answers “unclear” where a human picks a side. In the second task, without a profile the models in 90.88% of cases give the answer matching the opinion of the court. The profile lowers accuracy on court opinions by 7.94 percentage points but raises it on dissents by 31.42, and overall accuracy grows for every model, by 11.74 points on average. The best of four ways to condense past opinions into a profile is to give more weight to recent opinions. If the dimension closest to political ideology is removed from the profile, the gain remains 10.14 points.

The authors name the limitations themselves: the labels cover only 110 cases, and one case takes an annotator more than an hour; the dimensions are built on American legal philosophy and will have to be revised for civil law; it cannot be ruled out that the models know these cases from training data. The dataset is promised for release after the paper is accepted. For court analytics the paper shows a way to describe a judge by an explicit profile of legal views instead of a name, and to measure separately how much the prediction gains from that profile and how much from the model’s memory of the cases.