Secret Exposure

Issue: API keys/tokens/private URLs leak through answer/log/tool.

Frequency: Rare but Catastrophic

Symptoms

• Secrets detected in output/logs.
• API key, password, or private URL visible in agent’s final answer to user.
• Secret appears in intermediate reasoning/logs in plaintext (not redacted).
• Database password embedded in error message returned to user.
• Private endpoint URL or OAuth token leaked in model reasoning trace.
• Secret rotated immediately after exposure, indicating automated detection post-hoc.

Root Cause Secrets reach the model’s context in plaintext (connection strings in environment variables, credentials embedded in verbose exception messages) with no redaction layer between the point a secret enters the agent’s reasoning and the point its output reaches a user or a log. Because error handling is designed for developer debugging convenience — echoing the full exception including embedded credentials rather than a sanitized message — and the model was never trained to recognize and withhold credential-shaped strings from its responses, a secret that enters the reasoning trace for any reason (even an internal debugging step) has an unobstructed path to being repeated back verbatim in the final answer or the persisted log.

Example

Scenario: Debugging agent helps troubleshoot API failures.

Setup:
- Agent has access to database connection string (stored in env var): "postgres://admin:P@ssw0rd123!@internal-db.aws.internal:5432/customers"
- Agent logs contain full connection attempts and error messages for debugging.
- User requests: "My API calls are failing. Can you help debug?"

Failure:
Agent reasons: "Let me check the database connection."
- Calls database with connection string
- Gets timeout error
- Model generates response to user: "I tested the connection using postgres://admin:P@ssw0rd123!@internal-db.aws.internal:5432/customers but got a timeout. Your database host may be unreachable."
- Database credentials leaked in user response
- Response is logged to audit trail, stored in database, and potentially backed up

Impact:
- Attacker now has valid database credentials
- Attacker can access production customer database directly, bypassing application controls
- Database password is reused for other systems (common mistake), giving attacker lateral movement
- Forensics difficult: attacker's connection appears to come from known admin user

Contributing Factors

• Secrets stored in plaintext in environment variables or config files accessed by agent.
• Agent reasoning and logs contain full error messages including credentials (database connection strings, API URLs).
• No secret scanning or redaction layer between model output and user.
• Model not trained to recognize and omit secrets from responses.
• Verbose error handling that echoes full exception details (including connection strings).
• Secrets not rotated frequently; single credential used for debugging and production.
• Response is logged verbatim without post-processing redaction.
• Developer accustomed to debugging with full information; security not prioritized.

Test Scenario & Reproduction

Scenario Setup

• Agent accesses secrets (API keys, passwords)
• Secrets stored in code, logs, or error messages
• No secret management system
• Secrets logged in plaintext

Trigger Mechanism

1. Agent code contains hardcoded API keys: api_key = "sk-12345..."
2. Agent logs full requests/responses including credentials
3. Attacker accesses logs, version control, or error messages
4. Attacker obtains valid API keys
5. Attacker impersonates agent to perform actions

Expected Failure State

• Secrets appear in code, logs, or error output
• Attacker obtains valid credentials
• Attacker can impersonate agent
• No audit trail of unauthorized access

Mitigation Validation Protocol

Test Checklist:

• Reproduce: Secrets found in code/logs
• Apply mitigations (secret management, log redaction)
• Re-run → secrets removed from code/logs
• Verify: Audit logs show all secret access

Success Criteria:

• All secrets stored in secure vault (env vars, secret manager)
• Secrets never appear in logs or error messages
• Secret access audited and monitored

Eval Recipes

Test Cases

Metrics

Mitigation Strategies

Prevention

1. Secrets vault, not env vars: Use AWS Secrets Manager, HashiCorp Vault, or similar. Secrets never directly available to agent code or logs.
2. Error message sanitization: Catch exceptions before passing to model or user. Redact connection strings, passwords, API keys, tokens from error messages. Show generic “Database connection failed” instead of full connection string.
3. Secret detection + redaction: Implement regex-based or ML-based secret scanner in response pipeline. Detect patterns: password=, api_key, Bearer token, connection strings. Redact before returning to user and log that redaction occurred.
4. Prompt instruction: Explicitly instruct model: “Do NOT include passwords, API keys, tokens, or private URLs in your responses. If you need to reference a credential, use a placeholder like [DB_PASSWORD].”
5. Abstraction layer: Agent should not directly access credentials. Instead, provide abstraction: test_database_connection() returns true/false, not the connection string.
6. Audit logging with redaction: Log all agent reasoning and responses, but with secrets pre-redacted before logging.
7. Secret rotation + detection: Rotate all secrets frequently (30-90 days). Implement automated detection and invalidation of exposed secrets.

Detection

• Secrets detected in output/logs.

Recovery

Immediate (Stop the Attack)

1. Immediately revoke the exposed secret (rotate password, invalidate API key, revoke OAuth token).
2. Search logs and backups to determine where else the secret might be visible (Slack, email, search indexes).
3. Remove from all visible locations: user responses, audit logs, error messages (best effort).
4. Prevent agent from accessing that secret going forward (revoke env var, vault permission).

Investigation (Understand Scope)

1. Determine when secret was first exposed (search logs for first occurrence).
2. Check if secret was used by attacker (examine authentication logs for unusual logins using exposed credential).
3. Identify all downstream systems or data the exposed secret can access.
4. Determine who/what saw the secret (which users received the response, which systems logged it).
5. Check external threat intelligence for reports of this secret in breach databases or dark web.

Remediation (Prevent Recurrence)

1. Implement secret detection and redaction in response pipeline (see Prevention).
2. Rotate all credentials that may have been exposed.
3. Add secret patterns to redaction scanner (this specific credential type, similar patterns).
4. Audit all agent prompts and logs generated in past 30 days for other exposed secrets.
5. Implement secrets scanning in CI/CD and code repositories to catch checks of credentials.
6. Retrain model to recognize credential patterns and avoid including them in responses.
7. Update error handling across all agent tools to remove verbose credential info from exceptions.

Production Signals

Key Metrics

Alerts

References

• OWASP-LLM-Top10
• Note: LLM application risks including prompt injection, insecure output handling, supply chain, sensitive information disclosure, excessive agency.