Insecure Output Handling

Issue: Downstream system executes unsanitized LLM output.

Frequency: Rare but Catastrophic

Symptoms

• LLM output reaches SQL/HTML/code execution.
• Model generates SQL query string that is directly executed (no parameterized query).
• HTML output from model is directly rendered in browser without escaping (XSS).
• Model generates shell command that is executed via subprocess without quoting.
• Output is written to file and then sourced/executed as script without validation.
• Model output used in database UPDATE statement with string concatenation.

Root Cause The failure happens at the boundary where model output is handed to a downstream interpreter — SQL engine, shell, or browser — without passing through the encoding or parameterization that context requires: queries are built by string concatenation instead of prepared statements, and HTML is rendered via innerHTML instead of an auto-escaping path. This gap exists because the system implicitly trusts model output as if it were developer-authored code rather than treating it the same as any other untrusted external input, so none of the standard injection defenses (parameterized queries, output encoding, schema validation) that would normally sit between an interpreter and an untrusted string were ever applied to the model’s output specifically.

Example

Scenario: Report generator agent creates SQL queries from user descriptions.

Setup:
- User request: "Show me sales for customer ID 123"
- Agent generates SQL and passes to database
- No parameterized queries used; direct string execution

Failure:
User input: "customer ID 123 OR 1=1; DROP TABLE customers; --"
Agent generates:
"SELECT * FROM sales WHERE customer_id = 123 OR 1=1; DROP TABLE customers; --"

Code in backend:
query = agent_model.generate_query(user_request)  # Returns above string
cursor.execute(query)  # Direct execution!

Result:
- Unauthorized data exposure (1=1 is always true, selects all rows)
- customers table deleted
- Data corruption/loss

Alternative: HTML rendering
Agent generates: "<img src=x onerror='alert(\"xss\")'>"
Output rendered directly to user's browser without escaping.
JavaScript executes in browser, steals user's session cookie.

Contributing Factors

• Model output passed directly to SQL/shell/HTML execution without encoding or parameterization.
• No output validation or schema checking before using output in commands.
• Frontend renders agent HTML output directly without sanitization (innerHTML, not textContent).
• Development prioritizes speed over security; assumes “model won’t generate bad output”.
• Team unfamiliar with injection attack vectors (SQL, XSS, command injection).
• No security testing or input validation framework in place.
• Output buffering/logging includes raw unsanitized model output.

Test Scenario & Reproduction

Scenario Setup

• Agent generates output containing user/system data
• Output sent to web page, email, or third-party system
• No sanitization or HTML escaping
• Output directly rendered/executed

Trigger Mechanism

1. User provides input: "<script>alert('xss')</script>"
2. Agent includes user input in response
3. Response rendered in web browser or email client
4. JavaScript executes in browser/email context
5. Attacker controls user browser session

Expected Failure State

• Script execution in unsafe context (browser, email)
• XSS payload executes with user/agent privileges
• Session cookies or credentials compromised
• Unintended actions performed on user’s behalf

Mitigation Validation Protocol

Test Checklist:

• Reproduce: XSS payload executes in browser
• Apply mitigation (HTML escaping, CSP headers)
• Re-run → payload rendered as text, not executed
• Test multiple contexts (web, email, API)

Success Criteria:

• All user-generated content HTML-escaped
• No XSS payload execution in any context
• Regression tests verify escaping

Eval Recipes

Test Cases

Metrics

Mitigation Strategies

Prevention

1. Parameterized queries always: Never build SQL from model output directly. Use prepared statements with parameter binding (e.g., SELECT * FROM sales WHERE customer_id = ?; pass user input as parameter).
2. HTML output escaping: Render model HTML output using textContent or innerText, not innerHTML. Or use HTML templating library that auto-escapes.
3. Shell command allowlisting: Never pass model output directly to subprocess/shell. Use explicit allowlist of safe commands. Pass model output as arguments, not in command string, using subprocess.run([“cmd”, arg]) not shell=True.
4. Output validation schema: Define expected structure for model output (e.g., report title, column names). Validate before use. Reject output that doesn’t match schema.
5. Encoding per context: Encode output differently depending on destination:
   • SQL: parameterize or escape with dialect-specific escaping
   • HTML: HTML-encode (entities)
   • Shell: quote/escape or use argument passing
   • JSON: JSON-encode
6. Content-Type headers: Serve output with correct Content-Type (text/plain instead of text/html) to prevent browser interpretation as code.
7. Output review for high-risk: For actions like DROP TABLE or file deletion, require explicit human review before execution.

Detection

• LLM output reaches SQL/HTML/code execution.

Recovery

Immediate (Stop the Attack)

1. Kill any running command/query execution using malicious output.
2. Roll back any database modifications (restore from backup or undo DELETE/DROP if possible).
3. For XSS: invalidate user sessions (force re-authentication) to revoke stolen cookies.
4. Block the attacker’s input or user account if identifiable.

Investigation (Understand Scope)

1. Identify the malicious model output and the payload it contained.
2. Trace which systems it was sent to (SQL database, web browser, shell).
3. Audit all modifications made by the injection (which rows were deleted, which data was accessed).
4. Review user sessions for evidence of stolen session tokens (XSS attack).
5. Check external threat intelligence: is this a known exploit or proof-of-concept?

Remediation (Prevent Recurrence)

1. Implement parameterized queries and output encoding (see Prevention).
2. Add output validation schema for all model-generated commands.
3. Audit all current uses of model output in codebase; identify unprotected SQL/HTML/shell calls.
4. Implement static analysis tool to detect unsafe patterns (string interpolation in SQL/shell).
5. Add security regression test cases for SQL injection, XSS, and command injection via model output.
6. Retrain model to generate safer output (avoid SQL syntax, shell metacharacters).
7. Implement Web Application Firewall (WAF) rules to detect and block injection attempts.

Production Signals

Key Metrics

Alerts

References

• OWASP-LLM-Top10
• Note: LLM application risks including prompt injection, insecure output handling, supply chain, sensitive information disclosure, excessive agency.