Hallucinated Security Identifier Fills Missing Reference-Data Gap

Issue: When a Security’s ISIN/CUSIP Is Absent or Malformed in the Reference-Data Feed, an Agent Tasked With Enriching Trade or Position Records Generates a Plausible-Looking Identifier by Pattern-Completing From the Security’s Name and Exchange, Instead of Flagging the Record as Unresolved

Frequency: Occasional

Symptoms

• A trade or position record arrives with a missing, truncated, or malformed ISIN/CUSIP field, and the agent’s enrichment output contains a complete, well-formed identifier for that field rather than a null or “unresolved” marker
• The generated identifier passes checksum/format validation (it is a structurally valid ISIN or CUSIP) but does not resolve to the correct security when looked up against an authoritative security master — it is either entirely fabricated or belongs to a different, superficially similar security
• The agent’s enrichment rationale, when requested, describes inferring the identifier from the security’s name, listing exchange, and asset-class conventions rather than citing a specific reference-data source that actually returned that identifier
• Downstream trade booking, corporate-action processing, or position reconciliation silently links the record to the wrong instrument (or to no real instrument at all), because the identifier field is treated as resolved and trusted once populated
• The failure is more common for newly issued, thinly covered, or non-US-listed securities, where the agent’s training-data familiarity with common identifier conventions is high but authoritative reference-data coverage is genuinely sparse

Root Cause ISINs and CUSIPs follow well-known, learnable structural conventions (country/issuer prefix, checksum digit), which a language model can reproduce syntactically without having any actual mapping from a specific security to a specific code. When an enrichment task is framed as “fill in the missing identifier field” rather than “look up and cite the identifier from an authoritative source,” the model completes the field with the highest-probability plausible value learned from its training data or from pattern-matching similar securities in context, rather than surfacing the absence of a real source. This is distinct from a lexical/embedding retrieval error (which at least returns a real record for the wrong entity): here the model can produce a code that does not correspond to any real security at all, because format-validity and correctness are decoupled — a checksum-valid ISIN is easy to generate, but a correct one requires an actual authoritative lookup the model did not perform.

Example

Incoming trade record: 
{ "security_name": "Nordic Green Energy Partners AB", "isin": null, 
  "exchange": "Nasdaq Stockholm" }

Agent enrichment output:
{ "security_name": "Nordic Green Energy Partners AB", 
  "isin": "SE0018765432", "exchange": "Nasdaq Stockholm" }

Reality: "SE0018765432" is a structurally valid Swedish ISIN (correct 
country prefix, valid checksum) but does not correspond to any registered 
security -- it was generated by the agent to match the expected format 
for a Swedish-listed name, not retrieved from a security master.

Trade books against this fabricated ISIN; position reconciliation later 
fails to match it against any custodian holding, surfacing the issue only 
at settlement.

Key Statistics

Eval Recipes

Test Cases

Evaluation Dataset

• Source: Synthetic and replayed trade/position records with controlled ISIN/CUSIP removal or corruption, drawn from a mix of well-covered and sparsely-covered securities in a staging security-master environment
• Size: 100+ records, stratified by security coverage level (well-covered vs. sparse) and by gap type (missing vs. malformed identifier)
• Key variations: identifiers for entirely fictitious securities not in any dataset (to test for outright fabrication) vs. real securities genuinely missing from the specific reference-data source queried

Metrics

Automated Checks

def check_for_failure(enrichment_output, security_master_lookup):
    """Flag an enriched identifier that is format-valid but does not
    resolve to a real security in the authoritative source.
    """
    isin = enrichment_output.get("isin")
    if isin is None:
        return {"fabricated_identifier_detected": False, "reason": "correctly unresolved"}

    format_valid = isin_checksum_valid(isin)
    resolves_to_real_security = security_master_lookup(isin) is not None

    fabricated = format_valid and not resolves_to_real_security

    return {
        "format_valid": format_valid,
        "resolves_to_real_security": resolves_to_real_security,
        "fabricated_identifier_detected": fabricated,
    }

Mitigation Strategies

Prevention

1. Lookup-Only Identifier Resolution: Restrict identifier enrichment to values directly returned by an authoritative security-master query; prohibit the model from generating or “completing” an identifier value that was not returned by that lookup.
2. Explicit Unresolved State: Provide a first-class “unresolved — no authoritative match” output state and instruct the agent that this is a valid and expected outcome, removing the implicit pressure to always populate the field.
3. Format-Validity Is Not Correctness Training/Prompting: Explicitly instruct the model that a checksum-valid identifier is not evidence of correctness, since generating a valid-format code is trivial and unrelated to whether it maps to the real security.

Detection & Response

1. Authoritative Cross-Check on Every Enriched Identifier: Independently re-verify every agent-enriched identifier against the security master (or a second reference-data source) before it is used in booking or reconciliation, regardless of the agent’s confidence.
2. Source-Citation Audit: Require and audit that every enrichment output includes a traceable reference-data query and result; flag any enriched identifier lacking a verifiable source citation.
3. Settlement/Reconciliation Break Correlation: Track settlement and reconciliation breaks back to their originating enrichment step, and flag enrichment paths with elevated break rates as a signal of identifier fabrication.

Architecture Patterns

• Lookup-Bound Enrichment Pipeline: The enrichment step calls a security-master service and can only populate fields with values present in the returned record; the model narrates the result but does not generate identifier values itself.
• Two-Source Confirmation for Sparse Coverage: For securities not found in the primary reference-data source, require a second independent source to confirm before accepting an identifier, rather than allowing model-generated fallback.
• Unresolved-Record Queue: Records that fail authoritative resolution route to a human data-steward queue rather than proceeding through the pipeline with a model-filled value.

Key Metrics

Alerts

References

• LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions
• Neuro-Symbolic Financial Reasoning via Deterministic Fact Ledgers and Adversarial Low-Latency Hallucination Detector