Place Matters: hallucinations by jurisdiction

The paper Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queries tests whether the hallucination rate depends on place. The authors took scenarios from inquiries in family, housing, employment and traffic offence law, localized them for Los Angeles, London and Sydney, and gave the models’ answers to lawyers for labeling. The average hallucination rate was 0.45 for Los Angeles, 0.55 for London and 0.61 for Sydney; by area it was highest in housing law, 0.649, and lowest in employment law, 0.440. Consistency of answers under repeated generation correlated negatively with the hallucination rate. The authors do not establish the causes of the differences. For Lexometrica this is an argument to publish metrics by jurisdiction and area of law separately: an overall score hides the gap between well-represented and poorly represented rules.