Using a randomized field experiment, researchers discover that Wikipedia articles affect judges’ legal reasoning.
Mixed appraisals of one of the internet’s major resources, Wikipedia, are reflected in the slightly dystopian article “List of Wikipedia Scandals.” Yet billions of users routinely flock to the online, anonymously editable, encyclopedic knowledge bank for just about everything. How this unauthoritative source influences our discourse and decisions is hard to reliably trace. But a new study attempts to measure how knowledge gleaned from Wikipedia may play out in one specific realm: the courts.
A team of researchers led by Neil Thompson, a research scientist at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), recently came up with a friendly experiment: creating new legal Wikipedia articles to examine how they affect the legal decisions of judges. They set off by developing over 150 new Wikipedia articles on Irish Supreme Court decisions, written by law students. Half of these were randomly chosen to be uploaded online, where they could be used by judges, clerks, lawyers, and so on — the “treatment” group. The other half were kept offline, and this second group of cases provided the counterfactual basis of what would happen to a case absent a Wikipedia article about it (the “control”). They then looked at two measures: whether the cases were more likely to be cited as precedents by subsequent judicial decisions, and whether the argumentation in court judgments echoed the linguistic content of the new Wikipedia pages.
It turned out the published articles tipped the scales: Getting a public Wikipedia article increased a case’s citations by more than 20 percent. The increase was statistically significant, and the effect was particularly strong for cases that supported the argument the citing judge was making in their decision (but not the converse). Unsurprisingly, the increase was bigger for citations by lower courts — the High Court — and mostly absent for citations by appellate courts — the Supreme Court and Court of Appeal. The researchers suspect this is showing that Wikipedia is used more by judges or clerks who have a heavier workload, for whom the convenience of Wikipedia offers a greater attraction.
“To our knowledge, this is the first randomized field experiment that investigates the influence of legal sources on judicial behavior. And because randomized experiments are the gold standard for this type of research, we know the effect we are seeing is causation, not just correlation,” says Thompson, the lead author of the study. “The fact that we wrote up all these cases, but the only ones that ended up on Wikipedia were those that won the proverbial ‘coin flip,’ allows us to show that Wikipedia is influencing both what judges cite and how they write up their decisions.”
“Our results also highlight an important public policy issue,” Thompson adds. “With a source that is as widely used as Wikipedia, we want to make sure we are building institutions to ensure that the information is of the highest quality. The finding that judges or their staffs are using Wikipedia is a much bigger worry if the information they find there isn’t reliable.”
A paper describing the study is being published in “The Cambridge Handbook of Experimental Jurisprudence” (Cambridge University Press, 2022). Joining Thompson on the paper are Brian Flannigan, Edana Richardson, and Brian McKenzie of Maynooth University in Ireland and Xueyun Luo of Cornell University.
The researchers’ statistical model essentially compared how much citation behavior changed for the treatment group (first difference: before versus after) and how that compared with the change that happened for the control group (second difference: treatment versus control).
In 2018, Thompson first visited the idea of proving the causal role that Wikipedia plays in shaping knowledge and behavior by looking at how it shapes academic science. It turns out that adding scientific articles, in this case about chemistry, changed how the topic was discussed in scientific literature, and science articles added as references to Wikipedia received more academic citations as well.
That led Brian McKenzie, an associate professor at Maynooth University, to make a call. “I was working with students to add articles to Wikipedia at the time I read Neil’s research on the influence of Wikipedia on scientific research,” explains McKenzie. “There were only a handful of Irish Supreme Court cases on Wikipedia so I reached out to Neil to ask if he wanted to design another iteration of his experiment using court cases.”
The Irish legal system proved the perfect test bed, as it shares a key similarity with other national legal systems such as the United Kingdom and United States — it operates within a hierarchical court structure where decisions of higher courts subsequently bind lower courts. Also, there are relatively few Wikipedia articles on Irish Supreme Court decisions compared to those of the U.S. Supreme Court — over the course of their project, the researchers increased the number of such articles tenfold.
In addition to looking at the case citations made in the decisions, the team also analyzed the language used in the written decision using natural language processing. What they found were the linguistic fingerprints of the Wikipedia articles that they’d created.
So what might this influence look like? Suppose A sues B in federal district court. A argues that B is liable for breach of contract; B acknowledges A’s account of the facts but maintains that they gave rise to no contract between them. The assigned judge, conscious of the heavy work already delegated to his clerks, decides to conduct her own research. On reviewing the parties’ submissions, the judge forms the preliminary view that a contract has not truly been formed and that she should give judgment for the defendant. To write his official opinion, the judge googles some previous decisions cited in B’s brief that seem similar to the case between A and B. On confirming their similarity by reading the relevant case summaries on Wikipedia, the judge paraphrases some of the text of the Wikipedia entries in his draft opinion to complete his analysis. The judge then enters his judgment and publishes his opinion.
“The text of a court’s judgment itself will guide the law as it becomes a source of precedent for subsequent judicial decision-making. Future lawyers and judges will look back at that written judgment, and use it to decide what its implications are so that they can treat ‘like’ cases alike,” says coauthor Brian Flanagan. “If the text itself is influenced, as this experiment shows, by anonymously sourced internet content, that’s a problem. For the many potential cracks that have opened up in our “information superhighway” that is the internet, you can imagine that this vulnerability could potentially lead to adversarial actors manipulating information. If easily accessible analysis of legal questions is already being relied on, it behooves the legal community to accelerate efforts to ensure that such analysis is both comprehensive and expert.”
