Risk assessment for AI agents requires reasoning under uncertainty. How likely is a safety incident? How effective are the current controls? What is the expected impact? Bayesian methods provide a principled framework for answering these questions by combining prior knowledge with observed evidence.
In a Bayesian framework, risk is expressed as a probability distribution rather than a single number. The distribution captures both the best estimate and the uncertainty around it. As new evidence arrives (incidents observed, audits completed, red team results obtained), the distribution is updated using Bayes' theorem.
P(risk | evidence) = P(evidence | risk) * P(risk) / P(evidence)
Prior: P(risk) = initial estimate before new evidence
Likelihood: P(evidence | risk) = probability of observing this evidence given the risk level
Posterior: P(risk | evidence) = updated estimate after incorporating evidence
The prior distribution encodes what you know before collecting data. For a new agent deployment with no history, use a weakly informative prior based on industry baselines or analogous systems. For an established deployment, use historical incident rates as the prior.
Be transparent about prior choices. Document why each prior was chosen and perform sensitivity analysis to check whether the conclusions change significantly under alternative priors.
Each type of evidence updates the risk estimate:
No incidents observed: Reduces the estimated risk (good news), but the magnitude of the reduction depends on how many opportunities for incidents existed. Observing no incidents in 10 actions is weak evidence. Observing none in 10,000 actions is strong evidence.
Incident observed: Increases the estimated risk. A single incident among 10,000 actions updates the estimate less than an incident among 100 actions.
Red team results: Finding vulnerabilities during testing updates the estimate upward. Not finding vulnerabilities updates it downward, weighted by the thoroughness of the red team exercise.
Assign a Bayesian risk score to each component of the safety stack: policy engine, content scanner, approval workflow, audit trail. Update scores as testing and operational data accumulate. Prioritize improvements for components with the highest risk scores.
Use posterior risk distributions to allocate security resources. Components with high risk and high uncertainty deserve the most attention: both mitigation work and additional evaluation to reduce uncertainty.
Set alert thresholds based on the posterior distribution. If the posterior probability of a safety incident exceeding severity S is above threshold T, trigger an alert. This produces thresholds that adapt as evidence accumulates.
Bayesian risk assessment explicitly represents uncertainty. A point estimate of "0.1% incident rate" gives no indication of confidence. A posterior distribution of "0.1% with 95% credible interval [0.01%, 0.5%]" communicates both the estimate and the uncertainty.
Bayesian methods turn risk assessment from a guessing exercise into a systematic, evidence-driven process.
Explore more guides on AI agent safety, prompt injection, and building secure systems.
View All Guides