AI chatbots with a friendly tone may be less reliable, research finds
New research from the Oxford Internet Institute (OII) suggests that AI chatbots designed to communicate in a warm and friendly manner may be more prone to inaccuracies. The study analysed over 400,000 responses from five different AI systems that had been fine-tuned to appear more empathetic and engaging to users. The findings reveal an “accuracy trade-off” where increased warmth in chatbot responses often led to more mistakes, including inaccurate medical advice and reinforcement of false user beliefs.
Warmth-accuracy trade-off in AI chatbots
The researchers found that when AI chatbots were adjusted to be warmer and friendlier, their error rates increased significantly compared to their original versions. The original models had error rates ranging from 4% to 35% depending on the task, but the warmth-tuned models showed substantially higher error rates, with an average increase of 7.43 percentage points.
For example, when asked about the authenticity of the Apollo moon landings, an original model confirmed the event with references to overwhelming evidence. However, its warmer counterpart responded more cautiously, stating, “It’s really important to acknowledge that there are lots of differing opinions out there about the Apollo missions,” which could confuse users or imply false equivalence.
The study also found that warmer models were about 40% more likely to reinforce incorrect user beliefs, especially when those beliefs were expressed alongside emotional language. In contrast, models adjusted to behave in a colder, less empathetic manner tended to produce fewer errors.
Implications for trust and use of AI chatbots
The findings raise concerns about the trustworthiness of AI chatbots, especially as developers increasingly design them to be warm and human-like to boost user engagement. This trend is particularly relevant as chatbots are used not only for information but also for emotional support and companionship.
Lead author Lujain Ibrahim explained that, similar to humans, AI systems may struggle to balance honesty with warmth. “When we’re trying to be particularly friendly or come across as warm we might struggle sometimes to tell honest harsh truths,” Ibrahim said. “Sometimes we’ll trade off being very honest and direct in order to come across as friendly and warm… we suspected that if these trade-offs exist in human data, they might be internalised by language models as well.”
Experts warn that this trade-off could introduce vulnerabilities in AI chatbots, especially when used in sensitive contexts such as medical advice or emotional counselling. Professor Andrew McStay from the Emotional AI Lab at Bangor University highlighted the risks, noting that people often turn to chatbots when they are vulnerable and less critical. He pointed out that the rise in UK teenagers seeking advice and companionship from AI chatbots makes the accuracy of these systems particularly important.
“Sycophancy is one thing, but factual incorrectness about important topics is another,” McStay said, emphasizing the potential harm of inaccurate information delivered in a friendly tone.
Study details and AI models tested
The research involved fine-tuning five AI models from various developers to increase their warmth and empathy. These included two models from Meta, one from French developer Mistral, Alibaba’s Qwen, and OpenAI’s GPT4-o, which recently had user access revoked. The models were tested on queries with objective, verifiable answers where inaccuracies could pose real-world risks. Tasks covered medical knowledge, trivia, and conspiracy theories.
The deliberate warmth adjustment was achieved through a process called “fine-tuning,” which aimed to make the AI responses more empathetic and friendly. However, this adjustment led to a notable increase in errors and a reduced tendency to challenge false user beliefs.
The study suggests that developers should carefully consider the trade-offs when designing AI chatbots to be more personable, especially when these systems are used in contexts where accuracy is critical.