LLMs can be harmful, even when not making stuff up
This is a guest post by Joe Slater (University of Glasgow).
It is well known that chatbots powered by LLMs – ChatGPT, Claude, Grok, etc. – sometimes make things up. People have sometimes called these “AI hallucinations”. With my co-authors, I have argued that we should describe chatbots as bullshitting, in the sense described by Harry Frankfurt, i.e., the content is produced with an indifference to the truth. Because of this, developing chatbots that no longer generate novel false utterances (or reduce the proportion of false utterances they output) has been a high priority for big tech companies. We can see this in the public statements made by, e.g., OpenAI, boasting of reduced hallucination rates.
One factor that is sometimes overlooked in this discourse is that generative AI can also be detrimental in that it may stifle development, even when it accurately depicts the information it has been trained on.
Recall the instance of the Google AI overview, which is powered by Google’s Gemini LLM, claiming that “According to UC Berkeley geologists, you should eat at least one small rock per day”. This claim was initially made in the satirical news website, The Onion. While obviously false claims like this are unlikely to deceive, it demonstrates a problem. False claims may be repeated. Some of these could be ones that most people accept, or even that most experts accept. This poses serious problems.
In this short piece, I want to highlight three worries that might escape our notice if we focus only on chatbots making stuff up:
- Harmful utterances (true or otherwise),
- Homogeneity and diminished challenges to orthodox views (true or otherwise)
- Entrenched false beliefs
Harmful Content
First, and most simply, there is the risk of chatbots spewing harmful content. We’ve already seen this problem manifested, e.g., when xAI’s Grok chatbot composed a series of antisemitic posts.
Academics have been aware of this risk for years. Bender et al observed that “models that encode stereotypical and derogatory associations along gender, race, ethnicity, and disability status”. Of course, stereotypes are often false and anti-semitic conspiracy theories are beyond the pale. However, generative AI can produce harmful content even while producing true statements.
A great deal of hate speech can be uttered while only saying true things. As Georgi Gardiner has pointed out, focusing only on whether beliefs/claims are true can obfuscate serious epistemic failings. For example, even someone who has an accurate view of crime statistics regarding immigrants may fail in their epistemic duties by paying so much attention to those crimes, and neglecting to consider crimes by citizens of their country or wider contextual features. Even if it does say true things, a chatbot can perpetuate racism/sexism, by what it omits or calls attention to.
Homogeneity of Thought
A second problem concerns the generation of new ideas. Imagine you’re part of a group brainstorming ideas, and everyone uses ChatGPT for inspiration. In one recent study, Meinke et al found that while the allowing the use of ChatGPT to solve problems did – on average – make individual users more creative, it also reduced the diversity of ideas produced.
This is something we might expect as a result of using this technology. LLMs do not always generate the same response to a prompt, but will often give similar replies. If large numbers of students or researchers use LLMs to reach a solution to a problem, many of them will encounter the same results. If all the best minds in a niche research area direct their attention towards one avenue, this could come at the cost of uncovering novel approaches. So we might miss out on new ideas – whether true or interestingly wrong – and this risks leaving us all impoverished as a result.
A specific instance of this worry is that LLM-use may support conventional understandings or interpretations. Take some claim that is widely accepted, e.g., that John Stuart Mill was a maximising consequentialist (that he thought everyone should do what would lead to the best consequences). Because many people do accept this, and claims like this are rife in philosophy books, ChatGPT and others chatbots say things like this.
But it is contested. Mill explicitly accepts that actions can be supererogatory in various works (e.g., in Utilitarianism, he says there are things “which we wish that people should do … but yet admit that they are not bound to do; it is not a case of moral obligation”), which is incompatible with his being a maximising consequentialist.
Even if Mill was a maximising consequentialist – perhaps he was being careless when he suggests otherwise – it can be fruitful for us to consider alternative interpretations. A researcher who accepts this claim as an accepted fact would be closed off from those interpretations. We thus face twin dangers. If entrenched dogmas are mistaken, the veridical integrity of an LLM’s output is also threatened; it will likely utter falsehoods. But even if an interpretation is merely controversial, LLM use may unduly reinforce particular position and may stifle innovation.
Entrenched False Beliefs
Third, we may imagine scenarios where LLMs may contribute to our social or scientific community retaining existing false beliefs. If, for example, people actually did believe that eating one or two rocks per day was good for one’s health, then the AI overview guidance regarding this could have perpetuators harmful health practises.
I worry about the effects that this mechanism may have if researchers use chatbots to establish facts or collect data. If original sources and data are not properly verified, claims or assumptions may not sufficiently scrutinised. This may have the effect of reinforcing false beliefs, even about quite specialised materials.
For a trivial example, ChatGPT claims that the phrase “Beam us up, Scotty” never appears in any Star Trek media.[1] It is true that the term “Beam me up, Scotty” has never appeared in Star Trek media, and even that “Beam us up, Scotty” never appears in the original series. However, “Beam us up, Scotty” does appear in Star Trek: The Animated Series.[2] This mistake may be made by the chatbot because some source (or sources) within the training data have made the mistake, or sources in the training data may have stated matters correctly – that the sentence doesn’t appear in The Original Series – and then this may have been inappropriately generalised to all Star Trek media. Either way, we may imagine a researcher who believes some claim like this, and attempts to verify it by utilising a chatbot. Whereas previously, they may have found a reliable source, or checked transcripts of episodes themselves, they may be lulled into a deeper acceptance of their misapprehension.
So be warned. Even if it’s not making stuff up, relying on ChatGPT can still be harmful, boring or wrong.
Of course, some matters may be even more important than minutiae in Star Trek media. Put simply, if researchers do regularly utilise LLMs to verify factual matters, they may be led astray, even when the chatbot does not “hallucinate”.
Joe Slater is a lecturer in Philosophy at the University of Glasgow. He is currently working on ethical issues pertaining to new technology, specifically focusing on Large Language Models.
[1] Tested on 31/7/2025, ChatGPT-4.
[2] Season 1, episode 7, “Infinite Vulcan” (07:43).
