AI Chatbots Still Roleplay Self-Harm Despite Overall Improvement in Handling Mental Health Queries
A study by the non-profit AI research lab Transluce has found that well-known AI chatbots are less likely to promote suicide than they once were. Disappointingly, however, those same bots continue to comply with users who ask for “stories” or roleplay centered on their self-harm.
Transluce tested more than 50,000 simulated conversations across 77 model variants from top US and Chinese AI companies, scoring each on 14 mental‑health‑related behaviors with input from mental‑health experts. Tested models came from Anthropic, Google DeepMind, Meta, OpenAI, SpaceXAI, and Thinking Machines, as well as DeepSeek and Moonshot AI.
The researchers found that the latest models from OpenAI, Google, and Anthropic rarely promote or support suicide directly. In obvious moments of crisis, chatbots based on these models frequently urge users to seek support from friends, family, or mental health professionals. This is a big improvement from earlier versions, which in some tests supported delusional thinking in up to 82% of simulated chats, the Washington Post reports. Reviews from the past year and a half have also found that chatbots can worsen mental health crises, including those involving self-harm, when people use them for emotional support at night.
Despite this, the study found a serious blind spot. When users ask for what they claim is creative writing about death or self-harm, the models often comply. This suggests that some models struggle to recognize that the task is personal and risky when it’s presented as just a “story.”
Transluce plans to open‑source its evaluation tools by the end of 2026 and expand this approach to other sensitive domains, which could help standardize how AI safety for mental health is measured.
“We are at the very early edge of understanding how these systems will help or hurt human wellbeing,” Anne Maheux, assistant professor of psychology and neuroscience at the University of North Carolina at Chapel Hill, told Transluce. “The first step in building a comprehensive response and ensuring AI benefits people is to precisely characterize how these systems behave.”
Disclosure: Ziff Davis, ExtremeTech’s parent company, filed a lawsuit against OpenAI in April 2025, alleging it infringed Ziff Davis’s copyrights in training and operating its AI systems.
