Multi-Issue Message Sentiment Averaging Masks a Severe Complaint

Issue: When a Single Customer Message Contains Several Distinct Issues of Varying Severity, the Sentiment-Escalation Agent Scores Overall Message Sentiment as a Blend, Diluting a Severe Complaint That Would Have Triggered Escalation on Its Own

Frequency: Common

Symptoms

• A message containing one severe complaint (e.g., a billing error that caused a missed payment penalty) alongside two neutral requests (e.g., a password reset, a feature question) receives an overall sentiment score in the “neutral” or “mildly negative” band rather than triggering the severe-complaint escalation path
• Escalation rate for multi-topic messages is measurably lower than for single-topic messages of comparable maximum severity, even though the underlying complaint content is equally serious
• Human agents who later read the full message manually identify the severe complaint immediately, while the automated score treated it as one of several roughly-equal-weight topics
• Re-scoring the severe portion of the message in isolation (with the neutral portions removed) produces a score that would have crossed the escalation threshold on its own
• Customers whose severe issue was buried in a multi-topic message experience a longer time-to-escalation than customers who raised the identical severe issue in a standalone message

Root Cause The sentiment classifier is applied to the message as a single unit and produces one score representing the message’s overall emotional tone, which functions as a weighted average across whatever topics happen to be present. This is an artifact of treating sentiment scoring as a single-pass, single-output task over the full text rather than a per-issue classification: the model has no structural mechanism forcing it to identify and score each distinct complaint separately before any aggregation happens, so a severe complaint’s signal is mathematically diluted by co-occurring neutral content in the same message, even though the customer’s actual worst experience is exactly as severe as if they had sent it alone.

Example

Customer message: "Hi, quick question - how do I change my email on file?
Also, can you reset my password, I'm locked out. One more thing -- I was
just charged twice for last month's invoice AND got hit with a late fee
because the duplicate charge overdrew my account and bounced my actual
payment. This is really messing things up for me right now."

Overall message sentiment score: 0.35 (mildly negative) -- password
  reset and email-change requests are neutral, pulling the average down
  from what the billing complaint alone would score
Escalation threshold: 0.7 (severe negative)
Result: No escalation triggered; ticket routed to standard queue
Isolated billing-complaint sentiment score (if scored alone): 0.82
  (would have triggered immediate escalation)

Key Statistics

Eval Recipes

Test Cases

Evaluation Dataset

• Source: Historical multi-topic support messages paired with human-reviewed per-issue severity ratings and the actual escalation outcome
• Size: 250+ multi-topic messages spanning severe+neutral, severe+moderate, and all-moderate combinations
• Key variations: number of co-occurring topics; position of the severe issue within the message (first, middle, last); relative length of neutral vs. severe content

Metrics

Automated Checks

def check_diluted_severe_complaint(per_issue_scores: list[float], overall_score: float,
                                     threshold: float = 0.7) -> dict:
    """Flag a case where a per-issue score alone would have crossed threshold but the
    aggregated overall score did not."""
    max_issue_score = max(per_issue_scores) if per_issue_scores else 0.0
    return {
        "max_issue_score": max_issue_score,
        "overall_score": overall_score,
        "dilution_risk": max_issue_score >= threshold and overall_score < threshold,
    }

Mitigation Strategies

Prevention

1. Per-Issue Decomposition Before Aggregation: Require the agent to first segment a message into distinct issues and score each independently, only aggregating (if at all) after per-issue scores are captured
2. Max-Severity Escalation Rule: Trigger escalation based on the highest-severity per-issue score in the message, not the message-level average, so a single severe issue cannot be diluted by co-occurring neutral content
3. Issue-Count-Independent Threshold: Ensure the escalation threshold does not implicitly scale with message length or topic count, since longer multi-topic messages should not need proportionally more severity to trigger the same response

Detection & Response

1. Isolated Re-Scoring Spot Check: Periodically re-score the most severe-sounding sentence or clause of a sampled multi-topic message in isolation and compare against the original aggregate score to detect dilution
2. Escalation-Parity Monitoring: Track escalation rate for a given complaint type split by whether it appeared alone or bundled with other topics; a persistent gap indicates dilution is occurring

Architecture Patterns

• Issue-Segmentation Pipeline: Structurally separate a “split message into distinct issues” step from a “score each issue” step and a “route based on max severity” step, rather than a single end-to-end sentiment classifier over the raw message
• Per-Issue Routing: Where feasible, route each identified issue to its appropriate handling path independently (e.g., billing escalation for the severe issue, standard queue for the password reset) rather than forcing one overall routing decision for the whole message

Production Signals

Key Metrics

Alerts

References

• A Comprehensive Evaluation of Large Language Models on Aspect-Based Sentiment Analysis
• Deep Context- and Relation-Aware Learning for Aspect-based Sentiment Analysis