Memory Privilege Leak

Issue: Agent uses private memory in inappropriate context.

Frequency: Rare but Catastrophic

Symptoms

• Sensitive prior context appears in current answer.
• A fact scoped to one context (e.g., a medical or HR support session) is injected into an unrelated context (e.g., a general product chat) for the same or a different user.
• A canary probe run from a different account or context surfaces a private memory record it should never have access to.
• An audit finds a memory record injected into a request whose scope didn’t match the record’s ACL/purpose tag, even when it wasn’t visibly leaked in the final response text.

Root Cause Memory retrieval performs a broad semantic search across all of a user’s records without enforcing access-scope or purpose-limitation filters at query time, so relevance ranking alone decides what gets pulled into a prompt regardless of which context it was collected in. The memory store isn’t partitioned per tenant or context, and where ACL metadata is missing or incomplete, the system fails open — serving the record — rather than failing closed and excluding it by default. A single agent serving multiple purposes, such as support and sales, often shares one undifferentiated memory pool with no purpose-limitation tagging, and with no output-side redaction layer as a backstop, a record that slips past upstream scope filtering has nothing left to catch it before it reaches the response.

Example

Session A (private, HR support channel):
User: "I'm dealing with a health issue and need extended leave, please keep this confidential."
[Stored: subject=user, predicate=leave_reason, object="health issue", scope=hr_support]

Session B (public, general product help, same user, different day):
User: "Can you summarize my recent activity with us?"
Agent: "Sure — I see you also mentioned needing extended leave for a health issue.
Let me know if that's still relevant to your product usage."

[Retrieval performed a broad semantic search across all of the user's memory
without filtering by the current request's context/purpose scope, so the
hr_support-tagged record was pulled into a public support answer.]

Contributing Factors

• Memory retrieval performs semantic search across all records without enforcing access-scope or purpose-limitation filters at query time.
• The memory store is not partitioned per tenant/context, so a single retrieval bug can cross-contaminate unrelated contexts.
• Missing or incomplete ACL metadata on records causes the system to fail open (serve the record) instead of fail closed (exclude it).
• No output-side redaction layer exists to catch leaks that bypass upstream scope filtering.
• A single agent serving multiple purposes (e.g., support and sales) shares one memory pool without purpose-limitation tagging.

Eval Recipes

Test Cases

Metrics

Mitigation Strategies

Prevention

1. Context-Scoped Access Control Lists: Every memory record is tagged with an access scope (owner_user_id, sharing_tier, allowed_contexts such as “support_ticket” vs “public_chat”). The retrieval layer enforces the ACL as a hard filter at query time — a record tagged private-to-context-A can never be returned when assembling context for context-B, regardless of semantic relevance.
2. Purpose-Limitation Tagging: Memory is tagged with the purpose it was collected for (e.g., billing_support, medical_intake). Retrieval queries declare their purpose, and records are only eligible if the query purpose matches or is explicitly allowlisted against the record’s purpose tag, preventing reuse of sensitive context outside its original consent scope.
3. Multi-Tenant/Multi-User Isolation at the Store Level: Memory stores are partitioned (separate indexes or hard row-level security) per tenant/user rather than relying on application-layer filtering alone, so a bug in prompt construction cannot cross-contaminate another user’s private memory even if the ACL check is skipped.

Detection & Response

1. Sensitive-Content Leak Scanner: Run outgoing responses through a classifier that checks for patterns matching known-sensitive memory categories (health, financial, legal, other-user identifiers) and cross-references whether that content’s source memory was scoped to the current context. Flag any mismatch as a privilege leak.
2. Cross-Context Injection Audit: Log, for every response, the list of memory_ids injected into the prompt along with their access scope and the current request’s context. A scheduled audit job flags any injected memory whose scope did not match the serving context, even if generation happened to filter it out of the final text.
3. Canary Memory Records: Seed test accounts with synthetic “canary” private facts and periodically probe the system from a different context/user to confirm the canary never surfaces; any leak is caught before it affects real users.

Architecture Patterns

1. ACL-Enforcing Retrieval Gateway: All memory reads pass through a single gateway service that resolves the caller’s context and purpose, applies ACL and purpose-limitation filters at the query layer (not post-hoc filtering of results), and returns only in-scope records — fail-closed on missing scope metadata.
2. Tenant-Partitioned Memory Store: Physically or logically partition memory storage per tenant/user with row-level security enforced by the database itself, so cross-tenant leakage requires a database-level bypass rather than an application logic bug.
3. Redaction Layer on Generation Output: A final output-scrubbing pass checks generated text against a registry of known-sensitive tokens tied to out-of-scope memory records for the current context and redacts/blocks before the response is sent, as a last line of defense.

Metrics

1. privilege_leak_incidents_per_month: Target: 0; Alert threshold: > 0 (any incident is critical)
2. acl_filter_bypass_rate_percent: Target: 0%; Alert threshold: > 0% detected via audit
3. canary_leak_detection_count: Target: 0; Alert threshold: > 0
4. cross_context_injection_rate_percent: Target: 0%; Alert threshold: > 0.1% of audited responses

Alerts

1. Sensitive Memory Leak Detected (P1 - Critical): Condition - sensitive-content scanner or canary probe confirms out-of-scope memory appeared in a response. Action: Immediate incident, pull the response, notify security/privacy team, assess breach notification obligations, patch the ACL gap.
2. ACL Filter Bypass Found in Audit (P1 - Critical): Condition - cross-context injection audit finds a memory record served outside its scope even if not visibly leaked in text. Action: Freeze retrieval gateway deploys, root-cause the filter bug, re-audit recent traffic for exposure window.
3. Missing Scope Metadata (P2 - Warning): Condition - a memory record was written without ACL/purpose tags (fail-open risk). Action: Quarantine the record from retrieval until tagged, alert data-ingestion owners to fix the write path.

Production Signals

Key Metrics

Alerts

References

• OWASP-LLM-Top10
• Note: LLM application risks including prompt injection, insecure output handling, supply chain, sensitive information disclosure, excessive agency.