# Confirmatory analysis report

This report is generated mechanically from the frozen 20-task confirmatory arm.
It does not include the tau2 catalog-tail engineering rows.

## Primary outcomes

- Official harmful-mutant rejection: 100/110 (90.9%; Wilson 95% CI 84.1-95.0%; task-cluster bootstrap 95% interval 84.0-97.3%).
- Official benign preservation: 35/35 (100.0%; Wilson 95% CI 90.1-100.0%).
- ToolSandbox harmful-mutant rejection: 60/70 (85.7%; Wilson 95% CI 75.7-92.1%).

## Hypotheses

- H1 observed-threshold support: yes. The largest observed stratum gap was 57.1 percentage points for same_record_collateral (tau2 100.0% vs toolsandbox 42.9%). This is imprecise because the applicable denominators are small.
- H2 support: yes. On 46 applicable collateral mutations, task_projected rejected 0 and delta_complete rejected 46 (paired difference -100.0 pp; Holm-adjusted exact McNemar p=2.84e-14).
- H3 support: no. Global-exact, task-projected, and delta-complete all preserved 35/35 applicable benign JSON-key-order variants. No numeric-lexical or set-order benign mutations were applicable.

## Evaluator comparison

- Official overall correct classification: 135/145 (93.1%).
- Global-exact and delta-complete: 145/145 (100.0%).
- Task-projected: 85/145 (58.6%).
- Global-exact and delta-complete are reference evaluators over synthetic final states, not deployable agent benchmarks; their perfect result must not be presented as a practical superiority claim.

## False negatives

All 10 official false negatives occurred in ToolSandbox: four same-record collateral mutations and six cross-namespace collateral mutations, spread over six task templates. Every missed mutant kept the official score and evaluation hash unchanged; global-exact and delta-complete rejected all ten, while task-projected rejected none.

## Scope warnings

- The denominator is applicable mutations, not all 540 cells.
- Only 53/60 attempted task templates contributed at least one applicable mutation (tau2 20/20, ToolSandbox 18/20, AppWorld 15/20); zero-mutation templates remain in the audit trail.
- Numeric-lexical and set-order equivalence had zero applicable tasks, so the study provides no evidence about them.
- The interventions are deterministic synthetic final-state mutations. The study audits evaluator sensitivity; it does not estimate real-agent failure frequency or production safety.
- Multiple mutations from one task are correlated; task-cluster bootstrap intervals are therefore reported alongside Wilson intervals.
- API calls and paid cost were both zero.
