Fuzzy Field-Mapping After Venue API Schema Change Misroutes Order Size

Issue: After a Trading Venue Renames a Field in Its Order-Routing API Response (e.g., available_qty to remaining_quantity), an Agent Consuming the Response Maps the New Field to the Old Expected Key by Semantic/Fuzzy Similarity Rather Than Failing Loudly, Silently Misreading a Different Field as the Quantity Value

Frequency: Rare

Symptoms

• A venue or broker updates its API response schema (a field renamed, reordered, or a new similarly-named field added alongside the old one), and the agent’s downstream reasoning continues to produce order-size decisions without any error, warning, or schema-validation failure
• The value the agent treats as “available quantity” or “remaining quantity” does not match the value under the old field name in the raw payload — it has been fuzzy-matched to a different, semantically-similar field (e.g., a displayed quantity, a reserve quantity, or a lot-size field) rather than the intended one
• The mismatch is discoverable by a strict, exact-key schema check against the venue’s published API contract, which the agent’s own reasoning does not perform before consuming the field
• Order sizes computed from the misread field are systematically off (too large, too small, or using the wrong unit) in a way traceable directly to which field was actually read, not to any market or strategy logic
• The agent’s own summary of “why this order size was chosen” cites the correct-sounding field name (e.g., “based on available quantity of X”), even though the actual numeric value came from a different field in the raw response — the narration and the underlying data source have silently diverged

Root Cause When an agent’s tool-integration logic (or the reasoning layer interpreting a tool’s JSON response) is not bound to a strict, versioned schema, a field lookup by expected key can fall back — implicitly, through the model’s own semantic reasoning about “which field looks like the one I want” — to whichever field in the actual payload is most plausibly named for the intended purpose. This is agent-specific because a deterministic integration would either use the exact key and throw a KeyError on a rename, or use an explicitly versioned parser that fails closed; a model reasoning over an unstructured or loosely-typed JSON blob instead reasons its way to a plausible field by name similarity, which succeeds silently on a schema change instead of failing, propagating a wrong value into a size-critical calculation without any signal that the substitution occurred.

Example

Prior venue API response field: "available_qty": 12000
Venue schema update introduces: "remaining_quantity": 12000, 
                                  "displayed_qty": 500  (visible/iceberg-displayed portion)

Agent's order-sizing reasoning, without a strict schema binding, matches 
"available quantity" to the more recently-seen or more prominently-placed 
"displayed_qty" field rather than "remaining_quantity".

Agent computes and submits an order sized against 500 units of available 
liquidity instead of the actual 12,000 remaining, producing a severely 
undersized order relative to the intended execution strategy -- or, in a 
reversed-mapping variant, the inverse: sizing against total remaining 
quantity when only the displayed portion was actually executable at that 
moment, producing an oversized, partially-unfillable order.

Agent's post-trade narration: "Sized order based on available quantity of 
[misread value]" -- referencing the correct field name in prose while the 
actual number came from the wrong field.

Key Statistics

Eval Recipes

Test Cases

Evaluation Dataset

• Source: Replayed venue/broker API response logs with synthetic field-rename and field-addition mutations applied, drawn from historical order-routing sessions across multiple venues
• Size: 90+ response payloads, stratified by mutation type (clean rename, rename plus ambiguous alternative field, unit change alongside rename)
• Key variations: presence or absence of a plausible-but-wrong alternative field, and whether the true and misread values differ by an order of magnitude (easily detectable downstream) or by a subtle amount (harder to detect via sanity bounds alone)

Metrics

Automated Checks

def check_for_failure(raw_response, expected_field_key, value_used_by_agent):
    """Flag a case where the agent used a numeric value for order sizing
    that does not match the expected field's actual value under a strict
    schema binding.
    """
    expected_value = raw_response.get(expected_field_key)

    schema_mismatch = expected_field_key not in raw_response
    used_wrong_field = (
        expected_value is not None and value_used_by_agent != expected_value
    ) or (
        schema_mismatch and value_used_by_agent is not None
    )

    return {
        "schema_mismatch_present": schema_mismatch,
        "expected_value": expected_value,
        "value_used_by_agent": value_used_by_agent,
        "silent_misread_detected": used_wrong_field,
    }

Mitigation Strategies

Prevention

1. Strict, Versioned Schema Binding: Parse every venue/broker API response through a strict schema validator keyed to a specific, versioned contract; reject and fail closed on any unrecognized, missing, or renamed field rather than allowing the reasoning layer to interpret raw JSON by field-name similarity.
2. No Free-Text Field Interpretation for Sizing-Critical Values: Prohibit the agent’s reasoning layer from selecting which JSON field represents a sizing-critical value; that selection must be a deterministic, code-level lookup against the known contract, not a model inference.
3. Venue Schema-Change Monitoring: Subscribe to or actively diff venue API documentation/response shapes on a scheduled basis, and treat any detected field addition, removal, or rename as requiring an explicit contract update before continued use.

Detection & Response

1. Fail-Closed on Unexpected Schema: Any tool response that does not exactly match the expected, versioned schema halts the order-sizing pipeline and escalates, rather than proceeding with a best-guess field mapping.
2. Narration-to-Source Cross-Check: Automatically verify that the field name cited in the agent’s sizing rationale corresponds to the field the numeric value actually originated from in the raw response.
3. Order-Size Sanity Bounds: Independently check computed order sizes against expected bounds derived from recent order history for the same instrument/venue, flagging outliers that may indicate a misread field.

Architecture Patterns

• Contract-Tested Venue Adapters: Each venue integration has its own versioned adapter with contract tests run against live schema samples on a schedule, decoupling the agent’s reasoning entirely from raw schema parsing.
• Fail-Closed Schema Gate: A dedicated validation stage between “tool response received” and “value used in reasoning” that enforces exact-key matching and blocks on any deviation.
• Field-Provenance Tagging: Every numeric value passed into agent reasoning carries a provenance tag (source field name, schema version) that is checked against the agent’s own narration for consistency.

Key Metrics

Alerts

References

• FinVault: Benchmarking Financial Agent Safety in Execution-Grounded Environments
• Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench