Debugging Sentinel Anomaly Detection

Sentinel monitors agent behavior using statistical methods to detect anomalies. When it produces unexpected alerts or misses genuine anomalies, the issue is typically in the configuration, baseline calibration, or threshold settings. This guide walks through systematic debugging.

Understanding How Sentinel Detects Anomalies

Sentinel uses two primary statistical methods:

EWMA (Exponentially Weighted Moving Average) tracks a smoothed average of a metric. When a new observation deviates significantly from the smoothed average, it triggers an alert. The smoothing factor (lambda) controls how quickly the average adapts.

CUSUM (Cumulative Sum) accumulates deviations from a target value. When the cumulative sum exceeds a threshold, it signals a sustained shift in the metric. CUSUM is better at detecting gradual drift than EWMA.

Problem: Too Many False Alerts

If Sentinel alerts constantly, the thresholds are too tight for the agent's natural behavioral variation.

Step 1: Export the baseline data that Sentinel calibrated against. Examine the variance of the monitored metrics.

Step 2: If the baseline was collected during an atypical period (low traffic, limited tool usage), recalibrate during a representative period.

Step 3: Widen the alert thresholds. For EWMA, increase the number of standard deviations required to trigger. For CUSUM, increase the decision interval (h parameter).

Problem: Missed Anomalies

If Sentinel fails to detect behavior you consider anomalous, the thresholds are too loose or the monitored metrics do not capture the relevant behavior.

Step 1: Verify that the specific behavior is covered by a monitoring metric. If the agent is using tools in a new sequence but the same frequency, a rate-based metric will not detect it. You may need a distribution-based metric.

Step 2: Tighten thresholds gradually. Reduce the decision interval or the standard deviation multiplier in small increments.

Step 3: Add new metrics if existing ones are insufficient. Sentinel supports custom metrics that can track domain-specific behavioral signals.

Problem: Alert Storms After Deployments

When an agent is updated, its behavior changes deliberately. Sentinel interprets the new behavior as anomalous.

Fix: Reset the baseline after planned changes. Include a baseline recalibration step in your deployment process:

1. Deploy the update
2. Run in shadow mode for the calibration period
3. Reset Sentinel baseline
4. Enable full monitoring

Problem: Metrics Not Updating

If Sentinel metrics appear stale, verify that events are reaching the monitoring engine. Check that the control plane is forwarding audit events to Sentinel and that the AUTHENSOR_SENTINEL_ENABLED environment variable is set.

Document your threshold tuning decisions. When you change a threshold, record why, what the previous value was, and what behavior prompted the change. This history is invaluable when revisiting configurations months later.