Missing Data Mishandling in Financial Models

Issue: Model Handles Missing Data Incorrectly (Mean Imputation, Deletion); Introduces Bias or Information Loss

Frequency: Common

Symptoms

• Financial data has gaps (no trading on weekends, holiday closures)
• Model imputes mean price → Artificially smooth volatility
• Model deletes rows with missing data → Survivor bias
• Forward-fill creates lookahead bias

Root Cause Missing data is inherent in time series (no trading on holidays). Naive imputation (mean, forward-fill) has serious consequences. Mean imputation reduces volatility unrealistically. Forward-fill uses future data. Deletion introduces survivor bias. No unified “right” way; context matters.

Example

Scenario: Trading model trained on stock prices
Data: Monday-Friday trading prices
Weekend/Holiday: No trading (prices missing Saturday-Sunday)
Naive handling: Forward-fill (Friday price = Saturday price = Sunday price)
Result: Continuous time series but artificial (weekends have zero volatility)
Model learns: "Volatility is lower than reality"
Forward testing: Actual weekend returns volatile (market opens with gap)
Impact: Model underestimates risk; position sizing too aggressive

Key Statistics

• Data completeness: 80-95% typical (weekends, holidays missing)
• Volatility reduction by mean imputation: 10-30%
• Lookahead bias from careless forward-fill: 1-3% annual return bias

Mitigation Strategies

Prevention

1. Implement multi-layer entity resolution with hierarchy validation: Maintain a master entity reference database with parent-subsidiary relationships, guaranteed updater, transaction account mappings. Use persistent unique identifiers (LEI, ISIN, internal ID) instead of name-based matching. On every exposure lookup, resolve through hierarchy graph and validate against current regulatory filings. Root cause: Ensures exposure always attributed to correct legal entity accounting for corporate structure changes.

2. Establish regulatory compliance gates with before/after checks: Before any trading decision or exposure update, verify: (1) Counterparty regulatory status (sanctions check, credit rating current), (2) Position size vs. single-name concentration limit at ultimate parent level, (3) Exposure vs. concentration risk limits across correlated counterparties. Abort if any gate fails. Root cause: Prevents trades that violate compliance rules by checking compliance before execution.

3. Implement market data freshness validation with latency bounds: Every market data feed includes timestamp. Before using data for decisions, verify: (1) Timestamp within acceptable age (e.g., <30s for prices, <1d for ratings), (2) Data not marked as stale by upstream provider, (3) Cross-feed consistency check (e.g., bid-ask spread reasonable). Reject stale/inconsistent data with alert. Root cause: Prevents decisions based on outdated market information.

Detection & Response

1. Exposure aggregation audit with parent-level rollup: Daily batch job re-computes all exposure aggregations at ultimate parent level from scratch (not incremental). Compares against operational system. Flags: (1) Missing hierarchy mappings, (2) Exposure misattributed to legal entity instead of parent, (3) Concentration violations only visible at parent level. Reports with detailed reconciliation.

2. Regulatory compliance violation detection: Monitor all executed trades against post-hoc compliance checks. Flag violations: (1) Counterparty now in breach of sanctions/credit triggers after trade, (2) Concentration limit exceeded at parent level, (3) Position size violates regulatory limits for entity type. Generate audit trail for each violation with decision data.

Architecture Patterns

1. Corporate Hierarchy Graph Service: Maintains versioned parent-subsidiary-guarantee relationships. API: resolve_to_parent(entity_id, as_of_date) -> parent_id + risk_correlation. Fetches from regulatory filings (daily), M&A feeds (real-time), credit data (weekly). Triggers recomputation on family structure changes. Serves through cache with fallback to DB.

2. Pre-Trade Compliance Engine: Rule engine evaluates every proposed trade against: sanctions checks, concentration limits (computed at parent + correlated entities), regulatory position size limits, data freshness gates. Blocks non-compliant trades with detailed audit log of which rule failed why.

3. Market Data Freshness Orchestrator: Aggregates feeds from multiple market data providers with explicit ‘as of’ timestamps. Computes data freshness for each field (bid, ask, last_traded, credit_spread). Feeds below threshold age marked as ‘stale’. Risk system rejects decisions using stale feeds with incident log.

Key Metrics

Alerts & Escalation

References

• Missing Data in Financial Time Series
• Imputation Methods & Statistical Bias