Mean Time to Detect for AI Safety Incidents

Mean time to detect (MTTD) measures the average duration between when a safety incident begins and when it is detected. A low MTTD means threats are caught quickly, limiting the window for harm. A high MTTD means threats persist undetected, potentially causing significant damage before anyone notices.

Why MTTD Matters

The damage from a safety incident is roughly proportional to how long it persists. An agent exfiltrating data for 10 seconds leaks far less than one exfiltrating for 10 hours. Reducing MTTD directly reduces the potential impact of any incident.

Measuring MTTD

Calculate MTTD by tracking two timestamps for each incident:

• T_start: When the incident began (the first anomalous action, as determined during post-incident analysis)
• T_detect: When the incident was detected (the first alert, escalation, or human observation)

MTTD = average of (T_detect - T_start) across all incidents in the measurement period.

Benchmarks

There are no universal benchmarks for AI agent safety MTTD because the field is young. Set your own targets based on risk tolerance:

• Critical safety incidents: MTTD target under 5 minutes
• Policy bypass incidents: MTTD target under 15 minutes
• Behavioral anomalies: MTTD target under 1 hour
• Configuration drift: MTTD target under 24 hours

Reducing MTTD

Real-time monitoring: Authensor's Sentinel processes events as they arrive, not in batch. This enables detection within seconds of an anomalous action occurring.

Comprehensive coverage: Monitor all agent actions, not just a sample. Incidents that occur during unmonitored actions have effectively infinite MTTD.

Low-latency alerting: Ensure alerts reach responders quickly. A detection that sits in a queue for 30 minutes before being delivered adds 30 minutes to effective MTTD.

Reduced false positives: High false positive rates cause operators to delay investigation, increasing effective MTTD even when detection is fast.

MTTD as a KPI

Track MTTD as a key performance indicator for your safety program. Review it monthly. Investigate incidents with high MTTD to understand why detection was slow and what could improve it. Set improvement targets for each quarter.

You cannot fix what you cannot detect. MTTD measures how quickly you can start fixing.