Unit Testing AI Safety Policies

A policy rule that has not been tested is an assumption. Unit testing policies verifies that each rule produces the correct decision for specific inputs. Because Authensor's policy engine is pure and synchronous, policy unit tests are fast, deterministic, and require no external services.

Test Structure

Each test provides a policy, an action envelope, and an expected result. The test passes if the engine's evaluation matches the expected result.

import { PolicyEngine } from '@authensor/engine';

describe('data export policy', () => {
  const policy = {
    default_effect: 'deny',
    rules: [
      {
        action: 'data.export',
        conditions: { resource_classification: 'public' },
        effect: 'allow',
      },
    ],
  };

  const engine = new PolicyEngine(policy);

  it('allows export of public data', () => {
    const envelope = {
      action: 'data.export',
      resource: '/data/public/report.csv',
      metadata: { resource_classification: 'public' },
    };
    expect(engine.evaluate(envelope).effect).toBe('allow');
  });

  it('denies export of confidential data', () => {
    const envelope = {
      action: 'data.export',
      resource: '/data/confidential/users.csv',
      metadata: { resource_classification: 'confidential' },
    };
    expect(engine.evaluate(envelope).effect).toBe('deny');
  });
});

What to Test

Positive cases: Envelopes that should match an allow rule. Verify the action is allowed.

Negative cases: Envelopes that should match a deny rule or fall through to the default effect. Verify the action is denied.

Boundary cases: Envelopes at the edge of rule conditions. If a rule allows amounts up to 100, test with amounts of 99, 100, and 101.

Missing fields: Envelopes with missing metadata or incomplete conditions. Verify the engine handles them according to the fail-closed design.

Test Organization

Group tests by rule or by scenario. Rule-based grouping tests each rule in isolation. Scenario-based grouping tests realistic workflows that exercise multiple rules. Both approaches are valuable.

Coverage Metrics

Track which rules are exercised by tests. A rule without any test coverage is a blind spot. Aim for at least one positive and one negative test per rule. For rules with conditions, add boundary tests.

Running Tests

Since the policy engine has zero dependencies and is synchronous, policy tests run in milliseconds. Include them in the CI pipeline so every policy change is verified before deployment.

Testing Policy Changes

When modifying a policy, run the existing test suite first. If all tests pass, add new tests for the changed behavior. If tests fail, the change has broken an existing expectation, which may be intentional or may indicate a mistake.

Untested policies are undocumented assumptions. Tested policies are verified contracts.