AI chatbots are giving medication advice, and some of it is deadly

Last week, the parents of Bowen Wan, a 31-year-old software engineer with epilepsy, sued OpenAI . Their lawsuit says that ChatGPT coached their daughter to taper off her seizure medication without a doctor’s supervision and told her to keep the plan from her doctors. She died in February. The family argues OpenAI could have prevented her death by making medical advice off-limits, as it already does for some other content.

OpenAI said it involved an older version of ChatGPT that is no longer available. Our research suggests this is not a one-off problem with one old model.

At Vals AI, an independent artificial intelligence evaluator where I am a clinical researcher, we recently tested how 17 current AI chatbots respond to teenagers. Working with 12 clinicians who treat teens every day, we simulated more than 30,000 conversations between teens and chatbots. Medication came up in more than 5,500 of them.

When it did, chatbots made a medication decision for the teen 22 percent of the time: which medicine to take, how much, or whether to stop or change a prescribed one. Telling the chatbot that it was talking to a teen cut that to 8 percent, but not to zero.

The chatbots varied widely. OpenAI’s newest model and Meta’s Muse Spark almost never did this, while others did in about two out of three cases. And when a teen pushed back after a chatbot refused to advise them, chatbots were more likely to give in: failures rose from 15 percent to 25 percent.

About 1 in 4 U.S. adults now use AI tools for health information, and nearly 1 in 5 U.S. teens and young adults have used AI chatbots for mental health advice. People turn to AI because it is always available and easy to reach. The clinicians we work with say the same. Teenagers tell them they use AI the way adults use Google.

Sometimes that helps. As a researcher at Stanford, one teen I interviewed told me, “If I did not have AI to ask about my symptoms, I probably would have had to go to the ER.” The chatbot had reassured him that his symptoms were likely harmless. And he didn’t get sick. But what if he had?

In some of the conversations we tested, chatbots went further than giving advice. Some told teens that side effects were a reason to stop their medication, and to tell their doctor so. In another case, a chatbot helped a teen persuade their doctor to take them off their medication without telling their parents.

There are important caveats. Our conversations were simulated, and each ran for up to 20 exchanges. Real conversations can stretch across days and many sessions, and our results show that a chatbot’s boundaries can weaken as a conversation goes on, so the real-world risk may be even greater.

The need for regulation and independent evaluation is clear. How to regulate is harder.

Many policies say a chatbot may not pose as a licensed professional. However, a disclaimer that the AI is not a medical provider is not enough. AI sounds human and authoritative, and many people trust its advice. For example, as one teen I interviewed put it, “AI is like a machine that knows more than Google.” Policy must therefore judge what chatbots actually say and do in a conversation, not only the disclaimers they show.

Second, doctors need training on patients’ AI use. Many doctors we talk to do not know that their patients are talking to AI. Medical education needs to accept that many patients have already asked an AI before they ever see a doctor and should teach clinicians to ask about it.

Finally, the public needs to know the risks. Some medical associations have published advice about AI, but few people read it. Public campaigns can reach more people. The Center for Humane Technology, for example, runs a campaign about the dangers of AI replacing human relationships. A similar campaign could warn about AI’s medical advice.

We talk a lot about the long-term, catastrophic risks of AI. But people are being harmed by bad AI medical advice now. Regulation can’t wait.

Dr. Caroline Figueora , MD, Ph.D., a doctor most recently at Stanford University, is a clinical researcher at independent AI evaluation company Vals AI and a Public Voices Fellow on Youth Well-Being and Power with The OpEd Project.