Okay so yes it’s green text but…
I know people who work at inpatient acute facilities and they are constantly having to cockblock people to prevent truly horrible things from happening (think anorexia dating borderline or NPD + NPD)
So while this particular story may not have happened, this general story happens daily.
The LLMs are just somewhere between an averaging and a lossy compression of everything on GitHub. There’s nothing about the current paradigm of “AI” that is going to somehow do better than just rehashing that training set but with the inclusion of various classes of errors.
I think it’s better to view it as spicy search rather than any form of intelligence.
Worse yet a lot of mental health conditions have genetic components so the kids are basically getting the worst hand you can possibly deal them.