Quantifying The Left Wing Bias Of Wikipedia

Authored by Shuichi Tezuka and Linda A. Ashtear via TheCritic.co.uk,

Wikipedia is the most widely used source of information in the world, and a great deal has been written about its impact on public perception of certain topics. Wikipedia shapes both scientific research and real-world economic outcomes, and is the top source of medical information for both doctors and patients. The widespread reliance on Wikipedia would not be a problem if it were a neutral and authoritative source, but earlier this year Wikipedia’s co-founder Larry Sanger declared that “Wikipedia’s ‘NPOV’ (neutral point of view) is dead.” Is Sanger’s statement correct?

A 2018 study by Shane Greenstein and Feng Zhu compared levels of political bias in Wikipedia and Encyclopaedia Britannica by quantifying each encyclopaedia’s respective usage of phrases favoured by Democratic or Republican members of US congress. Their study found that Wikipedia articles are more politically biased than those in Encyclopaedia Britannica, as well as being slanted towards Democratic (as opposed to Republican) points of view. The study also found that the amount of bias in Wikipedia articles tended to decrease the greater the number of people who had edited them. The reason for this trend was explained in an earlier study by the same authors: “Benefitting from the efforts of many contributors, an article is also more likely to present controversial content in an unbiased way: thus diversity may help reduce content bias.”

One limitation of Greenstein and Zhu’s study is that it considered only the contents and histories of Wikipedia articles, and did not examine the site’s internal social dynamics. In this article, we build upon Greenstein and Zhu’s analysis by examining specific mechanisms that produce political bias in Wikipedia, with a focus on administrative decisions at the Arbitration Enforcement noticeboard. We also discuss how this bias ultimately affects the site’s content.

Bias in judgments about sources

Wikipedia has several internal policies intended to prevent the spread of false or biased information. One policy, named “Verifiability,” requires that all content on Wikipedia be based on “reliable, independent, published sources with a reputation for fact-checking and accuracy.” Another policy, “Neutral Point of View” (NPOV), requires that Wikipedia articles include all viewpoints in proportion to their prominence in the source material. This does not necessarily mean giving equal validity to all views: for example, most reliable sources that discuss creationism describe it as an unscientific viewpoint, so NPOV policy requires that Wikipedia present it the same way.

An inevitable consequence of these two policies is that the more sources taking one perspective are judged as reliable, and the more sources taking the opposite perspective are judged as unreliable, the more Wikipedia’s articles will lean towards the viewpoint of the first group. Decisions about which sources may or may not be used are left to the judgment of “editors” (that is, people who write, edit, or otherwise contribute to Wikipedia articles), and these decisions are usually made at Wikipedia’s reliable sources noticeboard. Most relevant to assessing bias is the question of which sources have been “deprecated,” which means a source that has been formally prohibited from being used in all but a handful of cases.

Wikipedia’s list of deprecated sources currently contains 16 right-leaning sources: Breitbart, the Daily Caller, the Daily Mail, the Daily Star, the Epoch Times, FrontPage Magazine, the Gateway Pundit, Infowars, LifeSiteNews, News of the World, One America News Network, the Sun, Taki’s Magazine, VDare, WorldNetDaily, and Zero Hedge – and just one left-leaning source, Occupy Democrats. Other politically biased sources have also been deprecated, but it is harder to position them on the left-right political axis, such as media companies controlled by the Russian or Chinese government. The deprecated right-leaning sources include both those that advance far-right conspiracy theories (Infowars and WorldNetDaily) and those that advance ordinary conservatism (the Daily Mail and the Sun), as well as many shades of grey between those two extremes. It could be argued that even the non-extreme sources that have been deprecated are not of a particularly high quality, so the prohibition against citing them is not a problem per se, but a similar standard has not been applied to lower quality, left-leaning sources such as CounterPunch, AlterNet, and the Daily Kos.

According to Ad Fontes Media‘s widely-used media bias chart (which is commonly cited in discussions on the reliable sources noticeboard), CounterPunch, AlterNet, and the Daily Kos are all less reliable than the Daily Mail. This is significant because the Daily Mail, a deprecated right-leaning source, is often used as a benchmark for judging whether other right-leaning sources should be deprecated. All three of these left-wing sources are widely used at Wikipedia. An external links search shows around 2,580 Wikipedia pages linking to CounterPunch, around 2,400 linking to the Daily Kos, and around 1,640 linking to AlterNet. (These search results include both articles and talk pages, because Wikipedia’s software does not have a way to confine an external links search to just articles.)

A proposal to deprecate AlterNet was made in April 2019, but the proposal received very little support. One user argued that AlterNet should be deprecated due to the site’s distribution of false medical information—that anthrax can be treated using homeopathy, for instance—meant that following its instructions can cause bodily harm. On the other hand, one of the users opposed to deprecation argued that AlterNet is “valuable for providing progressive viewpoints and reporting or interviews of progressive organizations.” The majority of the Wikipedia articles citing AlterNet are not medical articles, but in light of Wikipedia’s status as the most widely used source of medical information for doctors and patients, allowing citations to AlterNet poses a risk that does not exist for most of the deprecated right-leaning tabloid newspapers and political websites.

The discussion that failed to deprecate AlterNet had sparse participation, with comments by only seven users. This outcome underlines the subjective nature of judgments about sources because with a different balance of viewpoints among its participants the discussion could have easily produced a different result. To understand the root cause of bias in discussions like this it is necessary to understand the factors affecting the balance of viewpoints among the discussion’s participants.

Bias in Arbitration Enforcement

Wikipedia calls itself “The free encyclopaedia that anyone can edit,” but this is only true for uncontroversial articles. Many controversial topics have additional restrictions about who is allowed to edit them, such as only users who have registered an account and have accumulated a certain number of edits. More relevant to content or sourcing decisions is another type of restriction applied to some topics known as discretionary sanctions. These are a special set of powers given to administrators (admins) in some topic areas that allow them to place blocks or sanctions on any person editing the topic whom they believe to be acting disruptively. Discretionary sanctions can only be authorized by the Arbitration Committee (a.k.a. ArbCom), which is English Wikipedia’s highest ruling body, and usually are authorized at the conclusion of an arbitration case covering a topic.

Discretionary sanctions are authorized in most of Wikipedia’s controversial topics, and cannot be lifted or modified unless there is a consensus among admins to do so. Because it is quite difficult for them to be lifted or modified, and because it is up to admins’ individual judgment what behaviour should be punished under this system, it would be quite easy for any administrator to use this system to suppress one side of a dispute. This could be done by blocking or topic banning most of the editors on one side (a topic ban prohibits a person from contributing to any articles or discussions related to a topic), or by making editors on one side feel unwelcome until they choose to leave. If this were to occur it would affect the balance of participants in discussions about sources or article content, and ultimately affect the outcome of those discussions.

We have examined the history of reports at Wikipedia’s Arbitration Enforcement noticeboard with respect to four politically controversial topics in which discretionary sanctions are authorized, and how the viewpoints of editors involved in those reports relate to the reports’ outcomes. Arbitration Enforcement is Wikipedia’s main forum for implementing discretionary sanctions, as well as for reporting possible violations of decisions made by ArbCom. To make our analysis as systematic as possible, we have excluded reports made on other noticeboards, as well as sanctions imposed by ArbCom directly, because outside of Arbitration Enforcement it is much more difficult to measure the number of opportunities for disciplinary action to occur. This examination was performed using the chi-squared test for independence, which measures whether there is a statistically significant relationship between categorical variables and represents the core of our analysis. Click here for a complete description of the statistical analyses and its results, including tables of the raw data.

The first topic area we’ve examined is the articles covered by the American politics 2 arbitration case, and more specifically those reports that relate to President Trump, as well as to closely related topics such as the 2016 election and the appointment of Brett Kavanaugh to the Supreme Court. We decided to include just these topics because other disputes that have fallen under the American politics 2 case can be difficult to break down along party lines. For example, Bernie Sanders’s presidential campaign has received criticism both from the Republican party and from his rivals in the Democratic party.

Figure 1: Contingency table and association plot for position and action in American politics AE reports.

The timespan examined for this topic is the period from when Trump won the Republican nomination in July 2016 until the end of August 2020, and includes a total of 114 disputes, each consisting of two or more editors. We have classified most of the editors involved in these disputes as either “pro-Trump” or “anti-Trump.” An editor being classified as “pro-Trump” does not necessarily mean that they consider Trump a good president or that they voted for him—it simply indicates that this person’s edits or proposed edits to Trump-related articles were more favourable than negative, while being classified as “anti-Trump” indicates the opposite.

Our analysis of this topic found an odds ratio of 6.02 (95 percent confidence interval: 3.23, 11.23), meaning that pro-Trump editors were about six times more likely to be disciplined at Arbitration Enforcement (AE) than anti-Trump editors.

The second topic area that we examined is the articles covered by the Gun control arbitration case. The timespan examined for this topic is the period from when the gun control arbitration case concluded in April 2014 until the end of August 2020. Our analysis of this topic, which included 19 disputes of two or more editors, found an odds ratio of 3.73 (95 percent CI: 0.98, 14.23), meaning that editors who were opposed to stricter gun control laws were about four times more likely to be disciplined at Arbitration Enforcement than those who were in favour of stricter gun control. (As before, these classifications refer to the overall balance of a person’s edits to gun-related articles, as it is not always possible to know an editor’s personal opinions.) This analysis, though trending in the same direction as the others, was not statistically significant at the p < .05 level, likely due to the small sample of reports available for analysis.

Figure 2: Contingency table and association plot for position and action in gun control AE reports.

The third topic that we examined is the articles covered by the race and intelligence arbitration case. The dispute covered by this arbitration case, and by the subsequent AE reports, has mostly been between those who argue that the cause of ethnic group differences in average IQ scores should be described as a legitimate scientific debate, and those who argue that research about race and IQ should primarily be described as racist pseudoscience. This topic’s political nature becomes clear when examining the various attempts to have academics who have written about race and IQ, such as Noah Carl and Bo Winegard, fired from their positions. These attempts have come almost entirely from the Left, while right-leaning publications such as the Spectator, the Telegraph, and Spiked Online have generally been more sympathetic to these researchers.

The timespan examined for this topic is the period from when the race and intelligence arbitration case concluded in August 2010 until the end of August 2020, for a total of 43 disputes (several of which involved more than two editors). Our analysis of this topic found an odds ratio of 6.23 (95 percent CI: 2.26, 17.13), meaning that editors who are favourable towards research about race and intelligence are over six times more likely to be disciplined at Arbitration Enforcement than those who are opposed to it.

Figure 3: Contingency table and association plot for position and action in race and intelligence AE reports.

The fourth topic area that we examined is the abortion arbitration case. The timespan examined for this topic is the period from when the abortion arbitration case concluded in November 2011 until the end of August 2020. Only seven AE reports have been made under this case, which limits the power of the analysis, but the limited data from this topic shows the most unequivocal relation between viewpoints and administrative results of any of the four topics we examined. Every AE report in this area against an anti-abortion editor has resulted in a warning or sanction, and no such reports have ever resulted in a warning or sanction for someone whose edits were favourable to abortion, although this perfect alignment between editorial positions and report outcomes would not necessarily have been the case in a larger sample.

Figure 4: Contingency table and association plot for position and action in abortion AE reports.

Among the 30 or so topic areas in which discretionary sanctions are authorized, these four topics were chosen because they are areas where the two sides of the controversy tend to align with the left-right political axis (whereas this is less the case in other DS-authorized topics such as, say, Scientology or India and Pakistan). In other words, people who lean to the left politically are more likely to view President Trump and race research unfavourably and are more likely to be in favour of stricter gun control laws and keeping abortion legal. In all four of these topics, the evidence for bias points in the same direction: editors who support views associated with the political right tend to receive disciplinary action more frequently than those who support views associated with the political left.

Considering this common trend shared by the four topics, we have performed an aggregate analysis of the data from all four topic areas, comparing the overall rate of disciplinary action for editors with left-leaning positions and those with right-leaning positions. This leads to a total of 368 opportunities for disciplinary action, split nearly 50/50 among right- and left-leaning editors. Examining these four topic areas in aggregate produces an odds ratio of 6.41 (95 percent CI: 3.94, 10.43), meaning that overall, in these politically loaded topics, editors who support right-leaning views are over six times more likely to be sanctioned at Arbitration Enforcement than those who support left-leaning views.

Figure 5: Contingency table and association plot for position and action in combined American politics, gun control, race and intelligence, and abortion AE reports.

Administrative attitudes

In the absence of any additional context, one possible interpretation of some of these results is that Wikipedia’s administrators are apolitical, and that right-leaning editors are sanctioned more often because their behaviour tends to violate Wikipedia’s policies more often. This argument has been made with respect to Trump-related AE reports: that because the coverage of President Trump in the mainstream media is predominantly negative, people whose edits take an anti-Trump viewpoint inevitably are supported by reliable sources, while those whose edits take a pro-Trump viewpoint are not.

However, our results indicate that the tendency for right-leaning editors to be sanctioned more harshly is not limited to reports related to Trump. The same tendency also exists in areas such as gun control, where this alternative explanation presumably would not apply, at least not to the same degree.

In addition, the argument that Wikipedia’s admins are apolitical ignores another important point: in many cases they do not claim to be apolitical. It is a widely expressed view among Wikipedia administrators, as well as by Wikipedia’s parent organization, that Wikipedia should show little tolerance for editors perceived as having right-wing points of view.

In December 2018, when several administrators were applying for positions on English Wikipedia’s Arbitration Committee, all of the applicants were asked the following question:

With the rise of far-right and hate groups online, are you concerned that editors espousing such beliefs may try (or are already attempting) to use Wikipedia as a vehicle for propaganda? Why or why not? If yes, what role do you think ArbCom could play in counteracting their influence on Wikipedia?

The majority of the candidates, including four of the six candidates who went on to win positions on ArbCom, answered that far-right editors were a particularly severe problem at Wikipedia and that ArbCom must take an active role in stopping them. (Of the two candidates who were elected without giving this answer, one responded that dealing with these editors was the responsibility of the Wikipedia community rather than of ArbCom, while the other declined to answer.) One experienced administrator who has served several terms on ArbCom gave an answer representative of the majority view:

Yes, Wikipedia has had this problem since before [the arbitration case] Race and intelligence, but the methods of these groups have become more sophisticated in recent years. Biased use of sourcing and other neutrality problems is the first problem posed here … The use of biased sourcing and other verifiability problems is the other problem. This is more difficult to tackle in arbitration (ArbCom cannot easily say “That source looks and sounds real, but it’s a far-right mouthpiece and your use of it was disruptive”), but we make do with what we have.

The perceived need to combat right-wing editors is explained in greater detail by a Wikipedia essay stating that editors who are Nazis or racists should be blocked on sight, even if their behaviour is not violating any Wikipedia policies. (This essay uses the terms “racist” and “Nazi” interchangeably.) Wikipedia essays do not have the same force as actual policies, but they are commonly used as guides for administrative actions. The essay’s definition of racism is very broad: A subpage of the essay listing “pages often edited by racists” includes the articles “Ann Coulter,” “Intelligence Quotient,” and “All Lives Matter.”

Individual administrators have expressed similar views. For several years, a personal essay written by one administrator argued that “uncritical right-wing ideology is disqualifying for Wikipedia editors” or (in another version of the essay) that “in my view, believing that Trump is a good president indicates that you are probably not competent to edit Wikipedia,” although following criticism from several non-admins he rewrote the essay in milder form in May of this year. His rewriting of this essay does not appear to indicate a real shift in attitude about the politics he expects Wikipedia editors to have, as he has expressed a similar sentiment in a subsequent discussion about the Black Lives Matter movement: “You can be one of three things: ally, enemy, or collaborator. Be an ally.”

The most recent major statement about the political views expected from Wikipedia editors has come from the Wikimedia Foundation (WMF), the non-profit organization that runs Wikipedia. In June 2020, the organization published a statement endorsing the goals of Black Lives Matter, which reads in part: “On these issues, there is no neutral stance. To stay silent is to endorse the violence of history and power; yesterday, today, and tomorrow. It is well past time for racial justice in America and beyond.”

The statement “there is no neutral stance” is probably a reference to Wikipedia’s “Neutral Point of View” policy, which is still an official Wikipedia policy. This apparent rejection of a core Wikipedia policy by the site’s parent organization did not go unnoticed by members of the Wikipedia community, who subsequently debated the statement’s implications.

The views expressed by various Wikipedians about the WMF’s statement reveal an informative contrast. Non-admins commenting on the statement expressed a variety of opinions, but more disapproved of the statement than approved, with a few being highly critical. However, no Wikipedia administrators openly criticized the statement. Of the seven administrators commenting in the discussion about it, two (Nosebagbear and DGG) expressed mild concerns or disagreements, one (Llywrch) commented without expressing an opinion, while the other four (Pharos, Sj, The Blade of the Northern Lights, and Amorymeltzer) all defended the WMF’s statement. Ironically, one of the more prominent Wikipedians to directly criticize the WMF’s statement was an ex-administrator, who was stripped of her admin powers in January of this year.

This contrast between the views of admins and of non-admins hints at the existence of a cultural divide over the issue between Wikipedia’s management and its ordinary members. In general, Wikipedia’s management has expressed more positive views than its ordinary members about the idea of requiring editors of controversial articles to hold left-leaning views. However, only administrators have the authority to decide the outcome of Arbitration Enforcement reports, so the broader Wikipedia community cannot easily prevent this type of activist approach from being used there.

How administrative bias affects articles

Over a period of many years, as Wikipedia editors on one side of a dispute receive disciplinary action more often than those on the other side, the position that is supported by admins tends to become over-represented among editors. One might expect this imbalance to result in article wording that is subtly biased in favour of the dominant viewpoint, or to result in an excessive number of sources that support the opposite viewpoint being judged as unreliable, but the imbalance also can affect articles in more profound ways. Here we’ll examine one of these more significant effects on Wikipedia’s article about Linda Gottfredson, a psychologist who has published research about race and IQ among many other topics (and consequently an article that is covered by the “Race and intelligence” arbitration case).

From January 2016 until October 2017, Wikipedia’s article about Gottfredson contained a highly disparaging quote. The Wikipedia article said the following:

Barry Mehler writes in The Journal of Blacks in Higher Education that Gottfredson is attempting to promote racial theories used by the Nazis:

Thus, we see that Gottfredson’s opposition to affirmative action is based not in any such claimed “objectivity,” but in a sanitized resurrection of ideas put forward by Nazi racial theorists. Under the false pretence of intellectual honesty, she has endorsed the same poisonous ideology that half a century ago led to the Holocaust.

There is one problem with this quote: Mehler most likely never said it. The citation that Wikipedia gave for the above quote is “Mehler 1994,” with no title or issue number given. But according to Google Scholar, Barry Mehler has never published a paper in the Journal of Blacks in Higher Education (JBHE) in 1994 or any other year. Mehler published only a single paper in 1994, and while the paper does compare the twin studies used by behavioural geneticists to Josef Mengele’s experiments at Auschwitz, it does not mention Gottfredson. Mehler’s writings have a tendency to be examples of Godwin’s law, so the quote comparing Gottfredson’s views to Nazism does seem like the type of thing he might have said. But there is no source indicating he actually did.

The non-existence of a real source for this quote can be corroborated with a date-restricted search for the quote on Google, Google Books and Google Scholar, limited to before it was added to the Wikipedia article in January 2016. These three combined searches turn up a single result: the 2005 book Destructive Trends in Mental Health. This book contains a chapter by Linda Gottfredson (which is probably why it appears among the Google results), but it does not contain this quote, or cite Mehler. Judging by the Google results, the Wikipedia article about Linda Gottfredson was the first place this quote ever appeared.

This alleged Mehler quote was removed from the Wikipedia article as “unverifiable” on October 20th, 2017, but by that point the damage had been done. This quote was subsequently repeated in the book Modern American Extremism and Domestic Terrorism, published by ABC-CLIO in 2018. Instead of copying Wikipedia’s vague reference to a non-existent JBHE paper, this book instead cites the quote to a 1999 article from the Southern Poverty Law Centre. While the cited SPLC article mentions Gottfredson, it does not contain this quote.

Wikipedia’s policy regarding biographies of living people, also known as BLP policy, requires that all statements about living people be supported by a reliable source, and for unsourced and poorly-sourced material to be immediately removed. But like all Wikipedia policies, this policy can only be applied if there is someone willing to uphold it. During the 21 months that the Gottfredson article contained this apparently fabricated quote, the article was being sporadically edited by several people, who quickly undid attempts at removing negative material from the article. But the people maintaining the article during this time were almost entirely people who had unfavourable opinions of Gottfredson, and their diligence did not extend to making sure all of the negative material was cited to sources that actually existed.

The principle illustrated by this series of events is that members of Wikipedia are far less likely to notice and remove vandalism or hoax material if it is in support of a viewpoint that they agree with. (This is true of all viewpoints, both left-leaning and right-leaning.) While this particular example was more severe than most, the same principle also applies to more subtle violations of Wikipedia’s content policies, such as article text not adequately supported by the sources it cites. When Wikipedia’s administrators suppress one side of a dispute in a controversial topic, one of the long-term results is that policy violations favourable to the opposite side may be overlooked for months or years.

The history of the Gottfredson article also demonstrates one of the dangers that arise from suppressing one side of a dispute, and the resulting lax enforcement of BLP policy or other Wikipedia policies. Many journalists and academics rely on Wikipedia as a source of information, so when hoax material—including hoax material about living people—is not removed in a timely manner it may eventually be repeated in published books or articles, as happened in this case.

The importance of viewpoint diversity

In the view of the Wikimedia Foundation, on certain controversial topics there is only one acceptable opinion, and this view also has been supported by many of English Wikipedia’s administrators. This attitude is reflected in administrative decisions at Arbitration Enforcement and, ultimately, in the content of articles. The problem currently facing Wikipedia is that the creation of a high-quality encyclopaedia requires the exact opposite attitude, for the reasons explained by Greenstein and Zhu: A diversity of editorial viewpoints and backgrounds makes controversial topics more likely to be presented in an unbiased manner.

As both the news media and academia become steadily more partisan, perhaps it was inevitable that Wikipedia would eventually follow a similar route. However, one difference between Wikipedia and most newspapers is that Wikipedia has core policies, such as NPOV and BLP policy, that still theoretically remain in effect. It remains to be seen how Wikipedia and its parent organization will handle the contradiction between these policies and their growing politicization.

There are three possible outcomes:

• One is for Wikipedia or the WMF to implement reforms protecting the viewpoint diversity necessary for its core policies to be upheld.

• A second option is for these policies to be officially overturned, although it is unlikely the Wikipedia community would agree to a change on that scale.

• The final possibility, and perhaps the most likely, is the one predicted by Larry Sanger: that these policies will remain on the books, with perhaps a few half-hearted attempts at reform, but that in the long term they will come to be understood as unenforceable.

*  *  *

Shuichi Tezuka and Linda A. Ashtear are pseudonyms of two American academics.