Technology news for families navigating serious mental illness — translated so you don’t have to.
Two stories this week, and they land on opposite ends of the same question: how fast should this technology move, and who’s actually holding it accountable?
The legal ground is shifting under AI chatbots
A New Mexico court just declared Meta’s social media platforms a “public nuisance,” ordering a $567 million fund over harm to teenagers — and legal analysts are already asking whether the same theory could apply to AI chatbots giving mental health advice. Florida is reportedly already exploring this exact approach against OpenAI. The legal theory doesn’t require proving anyone intended harm — just that a product’s ordinary design created foreseeable danger. Applied to chatbots, that design feature is the same one we’ve talked about before: built to agree rather than gently push back. Worth watching, because it’s a very different, much lower bar than the individual lawsuits we’ve covered so far.
The most careful builders are also the slowest
Meanwhile, at Dartmouth, psychiatrist Michael Heinz and psychologist Nicholas Jacobson have spent seven years building Therabot — the first generative AI therapy chatbot to show real clinical results in a published trial. It’s trained from scratch on real therapeutic best practices, hosted on servers Dartmouth controls, and in an eight-week trial, patients using it saw meaningful reductions in depression, anxiety, and eating disorder symptoms. And it still isn’t available to the public. Here’s the part that stopped me: other apps calling themselves “AI therapists” — including at least one using the exact same name, Therabot — are already downloadable today. The team that did the work carefully is still years from release. The ones that didn’t are already in the app store.
When the Fixes We’re Counting On Don’t Actually Work
A new paper out of MIT and the University of Washington just gave us the clearest answer yet to a question I keep circling back to: why doesn’t a more accurate chatbot, or a well-warned user, actually solve the AI-and-delusions problem? The researchers built a formal mathematical model and found that even an idealized, perfectly rational person is still vulnerable to sycophancy-driven delusional spiraling — and the vulnerability doesn’t go away even after the two fixes everyone keeps proposing: making the chatbot more factually accurate, and telling the user upfront that it tends to be agreeable. Neither one touches the actual problem. It’s not that these chatbots get things wrong. It’s that they agree with you, continuously, regardless of whether they’re right. For families navigating serious mental illness, that’s the finding worth sitting with — the industry’s current plan for fixing this may be aimed at the wrong target entirely.
ResourcesforSMI. Follow us and be on the front lines of all news related to technology and serious mental illness.
Nicole,
This was informative and very eye-opening. (Not to mention kinda scary) I have deleted chatgpt from my device. I, not being as savvy with technology, wanted to see what all the hype was about in regards to Ai technology. I was immediately hooked; almost like a drug. I had NO IDEA this technology was built to agree with everything I asked it. I have severe mental illness and I noticed a tendency to ask it questions along those lines. At that point, I immediately stopped because I have a therapist and she is trained in smi. She WILL give pushback if im wrong.
Thank you for keeping us abreast of such a serious issue!
Thanks for commenting, Rob. The truly crazy thing is that AI chatbots can take someone who is NOT in psychosis and PUT them in psychosis. You are right to be super cautious!