When AI chatbots give bad medical advice, who assesses the damage?

As more and more people turn to AI chatbots for quick, 24/7 medical advice, a new perspective paper co-authored by faculty at Binghamton University, State University of New York explores the risks this practice carries that are largely invisible to users – and healthcare systems.

The paper, written by researchers at Binghamton University, Stanford University, Texas A&M University, and Indiana University, and published in Nature Health, examines what happens when a chatbot gives someone bad health advice. The authors argue that the harm is hard to catch because the conversation stays on the AI company’s platform, the person has no place to report the problem, and outside researchers cannot measure the damage.

“We’re getting medical advice from these chatbots, but nobody—literally no one—is looking at this. If there’s an error or an issue, there’s just no way for people to know,” said Kaicheng Yang, an assistant professor in the School of Computing at Binghamton University’s Thomas J. Watson College of Engineering and Applied Science and one of the paper’s authors.

The researchers gave the example of a 60-year-old man who asked ChatGPT how to cut chloride from his diet and swapped table salt with sodium bromide. The man ended up being hospitalized with bromide toxicity after experiencing hallucinations and paranoia. This paper highlights that because the man got this advice from a chatbot and couldn’t retrieve the conversation, his doctors had no way to know exactly what he was told, making it nearly impossible to piece together what happened.

Because AI companies lack a transparency system for the visibility of the health advice they generate for patients, this paper argues that this built-in, systemic lack of visibility is a key aspect of AI-generated health information because it actively prevents oversight. “Harms are not only possible, but structurally hidden from the clinicians, researchers, and regulators who could otherwise detect and correct them,” the authors note.

This paper also emphasizes a key facet of chatbots generating answers to people’s health questions: they can be wrong. Chatbots can generate erroneous answers confidently, oversimplify information, or draw from outdated information/false claims. However, the authors also note that those risks can trickle down to those who are not actively seeking answers. With the prevalence of AI summaries in search engines and the integration of AI in social media platforms like X and Meta, you do not have to be actively searching for an answer to be given an incorrect bit of information.

“Sometimes I’m not even looking for health information, but just by browsing my social media feeds, it’s there. It just shows up, and we believe that could have undesirable outcomes, especially if medical misinformation or state actors are trying to manipulate the online discussion,” Yang said.

Whether a person is actively looking for answers or comes across them incidentally, the authors argue that the harm rarely leaves traces clinicians, regulators, or researchers can verify.

To combat this, the authors propose several methods to increase transparency and responsibility for AI companies. They note that AI companies should give users access to their own health conversations so they can share this information with clinicians and better investigate the pathway that led to a harmful event. They also recommend that companies develop a disclosure system that allows health guidance to be flagged, reported, and investigated.

They further recommend that social media companies take more responsibility by enhancing clear labeling of AI-generated health content, withholding that content until it is vetted by medical governing bodies, and similarly, that search engines should only utilize vetted information in summaries. Policymakers, the authors suggest, should also extend physician malpractice liability to AI chatbot companies.

“We do not think this is something we should rely on the companies to do, because their incentive is always to make more money,” Yang said. “Building such a system goes against that incentive. So we have to have some kind of third-party monitoring system, an independent evaluation.”

By David Hermanovitch  Binghamton University, State University of New York via Newswise. The Gayly online. 10/21/26 @ 6:10 p.m. CST.