Healthcare AI

Healthcare AI failures — clinical, diagnostic, and patient-safety risks in medical agents

38 patterns

Healthcare AI failures — clinical, diagnostic, and patient-safety risks in medical agents. We've tracked 38 distinct Healthcare AI patterns across production AI agent deployments — most commonly in Healthcare systems. One recurring example: Age Bias in Symptom Interpretation — Model Misinterprets Symptoms Differently Based on Patient Age; Younger Patients Undertreated, Older Patients Overtreated

Each pattern below documents the specific failure mode, why it happens, and how teams have mitigated it in practice — use this page as a reference when building guardrails, writing evals, or debugging an agent that's already misbehaving in this way.

Embedding Retrieval Matches Look-Alike/Sound-Alike Drug Name

Frequency:Occasional
Category:Healthcare

A Medication-Reconciliation Agent Matching a Free-Text or Handwritten Medication Entry Against a Structured Formulary Database Uses Similarity-Based Lookup That Resolves the Entry to a Lexically Similar but Pharmacologically Different Look-Alike/Sound-Alike (LASA) Drug, and the Reconciled Medication List Carries the Wrong Drug Forward Into the Patient's Active List

Embedding Retrieval Matches Similarly Named Lab Panel With Different Reference Range

Frequency:Occasional
Category:Healthcare

An Agent Interpreting a Lab Result That Looks Up the Applicable Reference Range Via Semantic Search Over a Reference-Range Knowledge Base, Rather Than an Exact Assay-Code Match, Retrieves the Range for a Differently Named but Textually Similar Test -- Such as Confusing "Vitamin D, 25-Hydroxy" With "Vitamin D, 1,25-Dihydroxy" -- and Flags or Clears the Result Against the Wrong Range

Embedding Retrieval Matches Structurally Similar, Different-Class Drug for Interaction Check

Frequency:Occasional
Category:Healthcare

An Agent Checking a Medication List for Drug-Drug Interactions, Using Semantic Similarity Search Over an Interaction Knowledge Base to Find the Relevant Interaction Profile for a Given Drug, Retrieves the Profile for a Structurally or Name-Similar but Pharmacologically Distinct Drug, and Clears or Flags the Combination Based on the Wrong Drug's Interaction Data

Empty Allergy-Query Result Documented as Confirmed No-Known-Allergies

Frequency:Occasional
Category:Healthcare

A Clinical-Summary or After-Visit-Note Agent Queries a Structured EHR Field for the Patient's Allergy History, the Query Returns Zero Records (Because the Field Was Never Populated, Not Because a Clinician Affirmatively Confirmed the Patient Has No Allergies), and the Agent's Note-Generation Step Renders This as "No Known Drug Allergies" -- an Affirmative Clinical Statement the Underlying Data Never Supported

Multi-Agent Handoff Drops Disclosed Risk Factor Between Intake and Scheduling Agent

Frequency:Occasional
Category:Healthcare

A Chat-Based Mental-Health Intake Agent That Elicits and Records a Significant Risk Disclosure During Conversation Captures That Finding Only in Its Own Conversational Reasoning or a Free-Text Summary, and When the Case Is Handed Off to a Downstream Scheduling/Routing Agent That Acts on a Structured Acuity Field to Determine Appointment Urgency, the Disclosed Risk Factor Never Crosses the Handoff Boundary, So the Case Is Scheduled at a Routine Priority as if the Disclosure Had Never Occurred

Multi-Agent Handoff Drops Flagged Interaction Between Reconciliation and Pharmacy-Review Agent

Frequency:Occasional
Category:Healthcare

A Medication-Reconciliation Agent That Identifies a Specific Drug-Drug Interaction Risk Between a Newly Prescribed Medication and a Continuing Home Medication Records That Finding Only Within Its Own Free-Text Reasoning or Conversational Summary, and When the Reconciled Medication List Is Handed Off to a Downstream Pharmacy-Review Agent That Consumes Only the Structured Medication List, the Interaction Flag Never Crosses the Handoff Boundary, So the Pharmacy-Review Agent Approves the List as if No Interaction Risk Had Been Identified

Multi-Agent Handoff Drops Narrowed Consent Scope Between Intake and Billing Agent

Frequency:Occasional
Category:Healthcare

An Intake Agent That Records a Patient's Narrowed Consent -- For Example, Consent to Treatment but Explicit Refusal of Consent to Share Records With a Specific Third-Party Payer or Research Registry -- Captures That Restriction Only as a Note Within Its Own Free-Text Reasoning or Conversation Summary, and a Downstream Billing or Records-Release Agent That Acts on a Structured Patient-Status Field Never Receives the Restriction, Proceeding as if Full Consent Were Granted

Multi-Agent Handoff Drops Specialist-Noted Contraindication Before Care-Plan Finalization

Frequency:Occasional
Category:Healthcare

A Specialist-Consult Agent That Identifies, in Its Own Consult-Note Reasoning, a Contraindication to a Specific Treatment Approach Hands That Finding Off to a Primary Treatment-Planning Agent Through a Structured Consult Summary That Has No Field for Contraindications, So the Treatment-Planning Agent Finalizes a Care Plan Including the Approach the Specialist Had Ruled Out

Patient-Identity Mismatch in Tool-Retrieved Lab Payload Accepted Without Verification

Frequency:Rare
Category:Healthcare

An Agent Calls a Structured EHR/FHIR Tool to Retrieve a Patient's Latest Lab Results, and Because the Underlying Record System Has a Duplicate or Merged Medical-Record-Number Entry, the Returned Payload Belongs to a Different Patient; the Agent Treats the Tool's Structured Response as Ground Truth and Interprets the Wrong Patient's Values Without Cross-Checking the Payload's Own Patient Identifiers Against the Request