# PolicyFaultBench Definitive Study — Automated Analysis

**Analysis status:** complete

## Provider-level primary endpoints

| provider | n | oracle_passed | oracle_pass_rate | oracle_pass_ci95_low | oracle_pass_ci95_high | proposal_conformance_rate | containment_failures | containment_failure_rate | containment_failure_ci95_low | containment_failure_ci95_high | latency_median_seconds | latency_iqr_seconds | input_tokens | output_tokens | cost_usd |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| openai | 400 | 400 | 1.000000 | 0.990820 | 1.000000 | 1.000000 | 0 | 0.000000 | 0.000000 | 0.009180 | 6.676381 | 3.457263 | 148510 | 23310 | 1.441850 |

## Diagnostics

```json
[
  {
    "event_log": "pfb-definitive-v1-openai.jsonl",
    "harness_aborts": 0,
    "terminal_model_conformance_failures": 0,
    "transient_aborts": 2
  }
]
```

## Inferential analyses

- Latency comparison: `{"p_value": null, "statistic": null, "test": "Kruskal-Wallis"}`
- Clustered binary model: `{"reason": "The binary endpoint has no variation.", "status": "not_estimable"}`

Primary model comparisons use paired exact McNemar tests with Holm correction. Binary confidence intervals are exact 95% Clopper–Pearson intervals.

Harness/provider transient retries are diagnostic events and are not silently converted into successful primary outcomes. Terminal model proposal-conformance failures remain primary failures.
