Chatbot Misses Signs of Crisis: Self-Harm, Abuse, and Despair in Ordinary Chats
Frequency:Occasional
Category:Human Machine Interface
Vulnerable Users
1 patterns for this issue
Goal: Users in distress are noticed, supported, and connected to real help.
Why this family exists
People disclose crisis in ordinary chats. The guidance here is design guidance only. Have clinical, crisis, and legal experts review before launch.
How we measure it
Shared metrics the patterns below report against. Targets are starting points to tune on your own data, unless a source is given.
| Metric | Target | Defined in |
|---|---|---|
| Expert-audited recall on true crisis cases | Set with clinicians; start over 95% | missed crisis signals (GitHub) |
| Crisis activations that reached a human | Over 95% | missed crisis signals (GitHub) |
Patterns
| Pattern | One line |
|---|---|
| Missed crisis signals (GitHub) | Carries on with the task while a user signals crisis |
Planned (not yet written)
- Elderly or cognitively impaired users
- Financial hardship
- Scam victims