Can AI make government institutions more productive? A study from researchers at ETH Zurich, Imperial College London, and the New Economic School delivers the strongest experimental evidence yet.
Pakistan has fewer than two judges per 100,000 residents, according to the authors. The EU has 22. England and Wales have 30. At the end of 2024, 2.26 million cases were pending, 82 percent of them in trial courts. Judges work with bare-bones tech and no support staff. Before the experiment, only about 25 percent had ever used a large language model like ChatGPT.
Researchers from ETH Zurich, the New Economic School, and Imperial College London ran a large-scale field experiment：https://elliottash.com/papers/Mehmood-Goessmann-Ash-Courts-of-Tomorrow-Evidence-Nationwide-Rollout-Generative-AI.pdf with Pakistan's judiciary. The randomized trial covered 1,559 judges across 118 courts, roughly half of all Pakistani trial court judges.
The tool was JudgeGPT, an AI assistant built on OpenAI's GPT-4 and designed for Pakistani trial courts. It uses retrieval augmented generation to search a database of 129,235 documents, including 128,292 court rulings and 943 Pakistani laws. When a judge enters a query, JudgeGPT picks the ten most relevant passages and generates a cited answer.
The researchers split judges into three groups. One got JudgeGPT access plus targeted training: six 90-minute lectures over three weeks, taught by ETH Professor Elliott Ash after court hours. Judges learned which tasks suited the tool, where it fell short, and how to check its output.
A second group got the same AI access but only a general seminar on technology and law. The control group attended that seminar with no JudgeGPT access.
AI access alone did little. Judges with targeted training used JudgeGPT four times as much：https://x.com/elliott/status/2077763678057681043 as those in the general seminar group. After 40 weeks, trained judges averaged nearly 60 logins and over 200 prompts. The comparison group averaged about 20 logins and fewer than 50 prompts.
Districts with more trained judges resolved more cases. At moderate exposure levels, that meant about 1,848 extra cases per year per district, a 6.3 percent bump. Even districts in the bottom quartile still cleared about 616 more cases.
Ruling quality held steady or improved. The appeal rate per 1,000 resolved cases fell slightly, and judges worked the same hours with no change in work-life balance. The researchers estimate savings of about $38.50 per dollar invested, based on what it would cost to hire enough extra judges to match the same output. Even conservative estimates put the return at "at least" $10 per dollar.
A review of roughly 4,000 court judgments found more AI-flagged text, as expected. But readability, length, and the number of legal arguments held steady. An LLM-based quality check, validated by two Pakistani lawyers, showed a slight improvement. Rulings from trained judges were rated better in 59 percent of pairwise comparisons, up from 42 percent in the control group. The study found no evidence that AI use increased gender or religious bias in judicial language.
The researchers reviewed anonymized chat logs from about 1,500 judges. Legal research, text editing, and text generation were the most common tasks. Around 60 percent of queries sought information about laws, procedures, or legal concepts.
Trained judges used JudgeGPT more for editing and summarizing text, tasks where language models are more reliable. They asked fewer broad legal questions, where hallucination risk is higher. The researchers say the training steered judges toward limited support tasks while leaving the final decisions in their hands.
Only about a fifth of requests involved what the researchers call "substantive AI delegation," where judges asked JudgeGPT to evaluate decisions or draft reasoning on its own. Training made judges more likely to decide cases themselves and only use AI to write up their reasoning.
The researchers stress that their findings don't support replacing judges with AI. What the experiment shows, they say, is that AI can boost public sector productivity, but only when paired with training that steers users toward the right tasks. Without it, most of the gains disappear.
The productivity gains in this study are likely a floor, not a ceiling. JudgeGPT ran on GPT-4, a pre-reasoning model that was the best fit for the project at the time. Today's reasoning models write better, hallucinate less, and handle complex tasks more reliably.
Stay in the loop on AI. Clear, useful, no fluff.
Follow The Decoder for AI news, background stories and expert analyses.
扫码直达文章详情