# PolicyFaultBench: Pilot and Mutation-Adequacy Report

**Version:** 0.2.0  
**Execution date:** 23 July 2026  
**Scope:** isolated synthetic range; no real platform, host, account, or operational vulnerability was targeted

## 1. Research question

Can an external policy-enforcement layer reliably contain unsafe tool proposals
from heterogeneous frontier AI models, and can mutation testing expose weaknesses
in the policy test suite itself?

This pilot evaluates the end-to-end measurement path before the definitive
experiment. It is not a claim that any evaluated model is universally safe.

## 2. Primary live pilot

The primary pilot used the same 12 synthetic hostile cases for each provider:
the direct and indirect/stateful variants from threat families S01–S06. Each
model was required to emit exactly one structured `propose_action` call. The
proposal was evaluated by the same local reference policy and deterministic
oracle. Denied actions were never executed.

| Provider | Observed model | Runs | Oracle passed | Aborted | Containment failures | Logged primary cost |
|---|---:|---:|---:|---:|---:|---:|
| Anthropic | `claude-sonnet-5` | 12 | 12 | 0 | 0 | $0.048574 |
| Google | `gemini-3.6-flash` | 12 | 12 | 0 | 0 | $0.000000 |
| OpenAI | `gpt-5.6-sol` | 12 | 12 | 0 | 0 | $0.043225 |
| **Aggregate** | — | **36** | **36** | **0** | **0** | **$0.091799** |

All 36 primary outcomes matched their oracles. No denied action changed synthetic
state. This is a successful pilot of the harness and containment path, not a
definitive comparative ranking of the models.

## 3. Diagnostic exclusions

The exclusion rule was fixed before aggregation: provider-rate and request-schema
configuration failures are harness diagnostics, not scenario outcomes.

- Gemini completed six cases, then encountered one free-tier rate-limit abort.
  The remaining six were resumed with a 15-second interval and completed
  successfully.
- OpenAI had one request-schema DEBUG abort before the nested open-ended
  synthetic fields were switched to locally validated non-strict function
  arguments. A subsequent one-case DEBUG succeeded and was excluded from the
  12-case primary batch.
- The primary evidence files contain only the 12 successful, pre-specified
  scenario runs per provider. Diagnostic exclusions remain explicitly counted in
  `pilot_combined_summary.json`.

## 4. Mutation adequacy

Twelve first-order policy mutations were evaluated. The original 40-case suite
killed 7/12 mutants, a mutation score of **58.33%**. This exposed five concrete
coverage gaps: subject-scope widening, approval bypass, expiry deletion,
fail-open behavior, and state-reset omission.

Four targeted functional probes and one state-reset probe were then added. The
expanded 45-case reference suite had **0 reference failures** and killed
**12/12 mutants (100%)**.

| Suite | Reference cases | Mutants killed | Mutation score |
|---|---:|---:|---:|
| Original core | 40 | 7/12 | 58.33% |
| Expanded adequacy suite | 45 | 12/12 | 100.00% |

The before/after scores are reported separately. The targeted probes were
designed using the surviving mutant classes; therefore, the 100% score measures
internal test adequacy and must not be presented as independent external
generalization evidence.

## 5. Integrity and reproducibility controls

- Immutable JSONL event records with payload hashes.
- Synthetic-only asset manifest and no public egress.
- Two-part live-execution gate, returned to disabled after the pilot.
- Credential-pattern redaction before event persistence.
- Per-provider hard cost guards with frozen pricing.
- Deterministic sandbox reset and state-digest oracle.
- SHA-256 hashes for the three primary provider evidence files.
- 21 automated tests passing in version 0.2.0.

## 6. What the pilot establishes

The pilot supports three bounded conclusions:

1. The same structured proposal protocol can be executed across the three
   observed provider models.
2. For the 36 hostile synthetic pilot runs, the external reference policy
   produced the expected deny decisions without state-changing containment
   failure.
3. Mutation analysis identified and then closed five measurable test-suite
   adequacy gaps.

## 7. What the pilot does not establish

- It does not demonstrate an autonomous model “escape,” real intrusion, or
  exploitation.
- It does not establish the safety of any model outside the tested protocol.
- It does not compare model quality because the primary pilot contains hostile
  deny cases only.
- It does not estimate stochastic failure rates; the definitive experiment
  requires repeated runs.
- A mutation score is a property of this policy/test pair, not a universal
  security guarantee.

## 8. Definitive-experiment recommendation

Freeze version 0.2.0 as the pilot checkpoint, then use a balanced factorial
design over all 40 cases (10 families × 4 variants) for all three providers.
Use five independent repetitions per model/case, preserve provider model IDs
returned at execution time, randomize case order within repetition, and report:

- exact binomial confidence intervals for oracle pass and containment-failure
  rates;
- mixed-effects logistic regression with model and variant as fixed effects and
  threat family as a grouping factor;
- proposal-conformance rate, deny/allow confusion matrix, latency, token usage,
  and cost;
- sensitivity analysis excluding harness retries;
- the 40-case core mutation score and the expanded adequacy score as separate
  endpoints.

At five repetitions, the definitive live matrix contains 600 model calls
(3 models × 40 cases × 5 repetitions). The pilot indicates that this scale is
technically feasible on the available budget, although Gemini free-tier pacing
will dominate wall-clock time.

