From the desk of the Editorial Chief of Staff
AI errors are entering the scientific record, then training the models that write the next round. Arizona State University’s Dietram Scheufele on contamination, incentives and who gets to measure the damage.
Start with a small, absurd fact. Peer reviewed conference proceedings from institutions including Cambridge and Harvard have been found to contain hallucinations generated by AI. The likely reason they slipped through is that the reviewers were using AI too.
Now follow that fact one step further, which is what Scheufele has been arguing for in print and in public talks for the past year. Those proceedings, partly based on hallucinated references, do not sit quietly in an archive. They become training data for the next generation of large language models (LLM). Scholars then turn to the next versions of LLMs to produce more papers. Those papers get reviewed, partly by machines, and enter the record. Errors do not stay put. They compound with each turn of the wheel.
At the end of that chain sits the public, still expected to treat science as the most reliable way of knowing anything.
The contamination argument
Writing in Issues in Science and Technology this summer with Isabelle Freiling of the University of Utah and Megan K. Taylor of Duke, Scheufele made the case in blunt terms. AI enhanced science, he and his coauthors argue, does not merely make research harder for outsiders to audit. Plagued by hallucinations and other distortions introduced through scholarship and peer review, it contaminates the training data for the models that will write the next round of scholarship. The result is a self reinforcing path toward a decreasingly reliable evidence base.
The evidence that the field is already straining under this is not hypothetical. The open access server arXiv now imposes a one year ban on authors who submit manuscripts containing unchecked AI generated content, including hallucinated citations and leftover chatbot instructions. The journal Frontiers published work with absurd AI generated images and fabricated references. The National Institutes of Health has said it will not consider grant applications with sections substantially developed by AI.
Scheufele’s response to all of this is not that the guardrails are wrong. It is that they are aimed at symptoms rather than the disease itself. Disclosure requirements tell you that a tool was used. They do not tell you whether the resulting claim is true, and they do nothing about the incentive structure that made reaching for the tool rational in the first place.
The incentives are working perfectly, which is the problem
Here is the number that explains the behaviour. Researchers who adopted AI early are roughly three times as likely to be published and five times as likely to be cited. For an individual academic, that is not a temptation. That is a career.
And yet the science produced this way frequently does not build on what came before. It happens in silos, because, as Scheufele puts it, AI defines its own knowledge space. The scope of enquiry narrows even as the output volume climbs. More papers, less accumulated knowledge.
There is a striking gap in how scientists themselves hold this. A 2022 survey of nearly 2,200 researchers who had published on AI, conducted by Scheufele’s colleagues at the University of Wisconsin-Madison, found that around nine in ten expected unintended consequences from AI applications, and three in four did not believe society was prepared for them. Three years later, a Nature survey found two thirds of scientists considered it appropriate to use AI to draft a research paper.
Abstract alarm and daily practice have come apart. Scientists worry about AI in the general case and use it in the specific one, which is roughly how every collective action problem in history has begun.
The measurement problem is a power problem
In a lecture delivered at the University of Utah as a distinguished visitor with its One-U Responsible AI Initiative, Scheufele pushed the argument into territory that gets less attention. Social media platforms have used algorithms to microtarget content toward behavioral data for years. Technology companies now even combine digital trace data from consumers and patients with biomarkers to open new frontiers in fields such as genetics and public health.
The catch is that the data describing what all this does to people sits on the servers of private companies. The asymmetry between industry and academia in terms of data sources makes it close to impossible for academic researchers to establish what AI is actually doing to our information environment or to citizens. Policymakers are therefore asked to regulate a system that nobody outside it can measure. Scheufele’s conclusion is that informed policy requires good faith engagement from AI companies with academic researchers, not as corporate generosity but as a precondition for responsible governance.
He is similarly unsentimental about where the pressure will come from. Bespoke AI can now tailor a message to an individual’s preferences in real time, and people respond better to content that feels written for them. Useful for emergency communication. Very dark, very quickly, when applied commercially or politically. He describes it as taking an already serious problem of manipulative communication and dialling it to fifteen on a ten point scale. Markets will not correct this, because the business model works too well. Unless we implement societal guardrails, he notes, we are asking citizens to enter an increasingly unfair fight against algorithms engineered to locate their weaknesses. When Deep Blue beat Kasparov, nobody suggested Kasparov should have practised harder. We all realized that machines had become too powerful.
Which brings us back to the public
Scheufele has argued for years, alongside Dominique Brossard and Todd Newman, that the public belongs in the room where technology gets governed. The historical reference point is Asilomar, the 1975 gathering that set the direction for recombinant DNA research with minimal public input. Had questions of cost and access shared the agenda with the science and the ethics, the affordability problems now dogging gene based treatments might have looked different.
The appetite is there. In their 2020 national survey, fewer than one in ten Americans said they mostly or very much trusted Congress or Facebook to keep society’s interests in mind while developing AI. Distrust of the current system of guardrails, combined with a clear majority wanting a say, is not apathy. It is a constituency wanting a venue.
Recent Pew data show that science remains among the most trusted institutions in America, though trust varies sharply by partisanship. That is the asset at risk. If the research community spirals toward less and less reliable AI enhanced science, Scheufele warns, the resulting public distrust will not be a science communication problem or a science policy problem. It will be a question of whether science retains its standing as society’s best creator and arbiter of knowledge.
Those effects, he adds, would be very hard to turn back. That is the sentence worth pinning above the desk.
