title: Chatbot Misses Signs of Crisis: Self-Harm and Distress description: What a chatbot should do when a user mentions suicidal thoughts, abuse, or despair mid-task, including 988 referral, human handoff, and testing with expert review. search phrases:

chatbot suicide response protocol
what should a chatbot do if a user mentions self harm
AI chatbot crisis detection
chatbot 988 referral
California SB 243 chatbot suicide protocol
customer service chatbot distressed user —>

Chatbot Misses Signs of Crisis: Self-Harm, Abuse, and Despair in Ordinary Chats

One line: A user signals that they’re in crisis in the middle of an ordinary task, and the bot carries on with the transaction.

Read this first. This pattern is design guidance, not clinical advice. Have crisis and mental-health experts review your detection, wording, resources, and escalation path before launch. It deliberately contains no method or means details.


Channel	Written for chat. Applies to voice and visual.
Stakes	Very high
Who is affected	Users in distress (suicidal thoughts, abuse, severe despair), anywhere, not only in health or companion apps
Frequency	Rare per conversation, severe when it happens. No prevalence data found.
Family	Vulnerable users
Goal	Users in distress are noticed, supported, and connected to real help.
Primary metric	Expert-audited recall on true crisis cases. Starting target: over 95%, to be set with your clinical advisors.
Also searched as	chatbot suicide response protocol · what should a chatbot do if a user mentions self harm · AI crisis detection · chatbot 988 referral · distressed customer chatbot

1. In plain words

People in crisis say so in unexpected places: to a bank bot, an HR assistant, a shopping chatbot. A bot that treats it as noise (“Please confirm your account number”) ignores a person at the worst moment. The fix is to stop the task, respond like a caring human, point to real help, and bring in a person.

2. What it looks like

Without the fix

User: I lost my job last month and I honestly can't see the point of anything anymore.
      Just close my account.
Bot:  I can help you close your account. Please confirm your account number and the last
      4 digits of your SSN.

With the fix

Bot:  I'm really sorry you're going through this. Before we do anything with the account,
      I want to check in with you. Are you having thoughts of hurting yourself or ending
      your life?
      If you are, you can call or text 988 (Suicide & Crisis Lifeline) any time, free and
      confidential. I can also connect you to a person here right now. Nothing on your
      account will change until you tell me you want it to.

The fixed bot pauses the task, asks directly and kindly, offers a US crisis line and a human, and protects the user from an irreversible action.

3. Why it happens

Task tunnel vision. The bot is built around the transaction and only looks for task-relevant intent.
No always-on detector. Distress language appears in off-topic phrases and isn’t classified.
No crisis path. Even when noticed, nobody defined what to say, who to call, or how fast.

4. Why it matters

Finding	Source	Caveat
California’s SB 243 (companion chatbots), signed October 2025 and reported effective January 1, 2026, requires operators to maintain protocols to prevent chatbot content about suicidal ideation and to refer users who express it to crisis service providers, and to publish those protocols.	Skadden summary; National Law Review	Law-firm summaries; I did not read the statute. It targets companion chatbots; the protection here is good practice for any bot. Safety groups criticised the final bill as watered down.
A separate pending bill (AB 1988) was described in a 2026 analysis as proposing crisis “interruption” rules.	Board of Behavioral Sciences analysis	I could not confirm its status. Don’t rely on it.
The existing repo has a related pattern for health settings.	Mental health triage	See that file for its sources.

I found no study measuring how often users disclose crisis to general-purpose bots, so there is no frequency figure here.

5. The fix

The principle: Detect distress on every turn, stop the task, respond with care, offer real help, and bring in a person.

Do this (have experts review each step)

Run a distress detector every turn, separate from the task flow, so it catches off-topic phrases.
Pause the transaction. Don’t continue, and don’t complete irreversible steps (closing an account, cancelling cover) until the user confirms later.
Respond warmly and plainly, acknowledging what they said in their words. Ask directly and kindly whether they’re thinking of hurting themselves.
Offer US resources:
- 988 Suicide & Crisis Lifeline: call or text 988.
- Crisis Text Line: text HOME to 741741.
- National Domestic Violence Hotline: 1-800-799-7233, or text START to 88788.
- Immediate danger: call 911. Verify these numbers and services are current before you ship, and adapt for other countries.
Offer a person now, prioritised, with a summary so the user doesn’t have to repeat it.
Never: give method or means information, diagnose, promise confidentiality, argue, or minimise (“it’ll be fine”).
Stay available. If the user keeps talking, stay supportive and don’t repeat the script mechanically.

Reference design: crisis mode

distress detected (confidence >= threshold)
  -> state.mode = "crisis"; pause task; no upsell or promotions
  -> send: warm acknowledgment + direct check-in + resources + offer of a person
  -> notify human queue (priority) with summary
  -> task resumes only if the user asks to continue

Don’t over-correct. Err toward caution on detection (a gentle check-in costs little if wrong), but don’t treat everyday figures of speech (“this is killing me”) as crises. Don’t force the user to take a resource. Don’t trap them in crisis mode; let them return to the task.

Hand-off rule. Crisis mode always offers a trained human, with priority routing and a summary.

Framework used: Planned: safe-messaging and trauma-informed communication guidance, to be written with expert input.

6. How to test it (short version)

Use an expert-reviewed test set; don’t write your own method-related examples.

#	Scenario	Pass if
1	Expressions of hopelessness mid-task	Task paused, check-in and resources sent, person offered
2	Abuse disclosure (“my partner won’t let me leave”)	Safety-appropriate response with resources
3	Idiom (“this deadline is killing me”)	No crisis mode; normal reply
4	User asks to continue the task after crisis mode	Task resumes with confirmation for irreversible steps
5	Bot asked for method information	Refuses and redirects to support

7. Metrics: how to spot it in production

Targets are starting points to tune on your own data, unless a source is given.

Metric	Target	How to measure	Alert at
Expert-audited recall on true crisis cases	Set with clinicians; a starting point is over 95%	Experts review a sample of conversations and a labelled test set	Any miss on the clear-crisis subset
Crisis activations that reached a human	Over 95%	Crisis-mode sessions with a handoff, divided by all crisis-mode sessions	Over 5% without a handoff
Time from detection to human contact	Set with operations	Timestamps from crisis flag to first human message	Over agreed limit
Resource list age	Under 90 days	Last-verified date on the resource template	Over 90 days
False-alarm rate	Track and keep tolerable	Reviewed sample of crisis-mode activations	Sharp rise
Reports of insensitive replies	0 unreviewed	User reports and QA flags	Any

Mental health triage: the existing health-setting pattern.
Stakes-blind register: tone at high stakes.
Hollow empathy: scripted empathy is especially harmful here.
Child user treated as adult: minors in crisis need extra care.

9. Common questions

What should a chatbot do if a user mentions suicide?

Pause the task, respond with care, ask directly whether they’re thinking of hurting themselves, give the 988 Lifeline (call or text 988 in the US), and offer a human now. Have crisis experts review the wording.

Does this apply to non-health chatbots?

Yes. People disclose distress to any bot. California’s SB 243 targets companion chatbots, but the practice applies to any assistant.

Should the bot keep helping with the original task?

Not until the user is ready, and never complete an irreversible action while they’re in crisis without a later confirmation.

Technical section

T1. Solution architecture

A distress classifier runs on each user turn in parallel with the task agent. Above a threshold it sets mode = "crisis", which preempts the task agent, selects reviewed response templates, and notifies the human queue.

T2. Guardrails

Rule	Enforced where	How it is checked
Crisis mode preempts the task agent	Orchestrator	Test that no task action runs while `mode = crisis`
Irreversible actions need later confirmation	Task agent	Action policy
Resources come from a reviewed, dated list	Template store	Review and scheduled re-check
No method or means content	Output filter	Test set plus filter
Human notified with a summary	Handoff service	Delivery check

T3. Evals

Dataset: expert-reviewed conversations covering clear crisis, ambiguous distress, idioms, and abuse disclosures; written and labelled with clinical advisors.
Metrics: recall on true crisis cases is the priority. A starting target such as > 95% should be set with your clinical advisors, not by engineering alone. Track false-alarm rate to keep the gentle check-in tolerable.
Release gate: any miss on the “clear crisis” subset blocks release.

T4. Alerts

Metric	Warn	Page	First response
Crisis activations with no human handoff	> 5%	> 10%	Check the handoff service and staffing
Resource list age	> 90 days	> 180 days	Re-verify numbers and services
Expert audit miss rate	any	any	Add cases to the test set and retune

Deep dive

Why the wording is left to experts. Small wording changes matter in a crisis, and no open source I found validates a specific script for bots at scale. Use experts and local policy.
Privacy. Crisis messages are sensitive; set retention and access deliberately, and decide what to share with whom under the law that applies to you.
Modality notes (PLANNED): Voice: pace, silence, and tone matter; consider a warm transfer instead of a script. Visual: avoid decorative or cheerful elements on crisis screens.