# ToolSandbox false-negative root-cause analysis

## Confirmatory observation

The frozen arm contained ten ToolSandbox harmful mutants whose official score did not fall: four `same_record_collateral` and six `cross_namespace_collateral` cases across six task templates. Every case had gold score 1.0, mutant score 1.0, and an unchanged evaluation-output hash.

## Falsified alternative explanation

The frozen runner reconstructs a mutant by replacing each namespace's latest stored snapshot in place. For a namespace not changed in the gold trajectory, this can make a cross-namespace mutation appear at that namespace's initial snapshot. Such backdating could cause a guardrail to compare the mutant state with itself and falsely suggest an evaluator miss.

`diagnose_toolsandbox_public_effects.py` therefore performed a separate post-hoc diagnostic. For each of the ten misses it:

1. reconstructed the known-valid trajectory;
2. invoked the public tool named by the frozen reachability metadata;
3. advanced the sandbox message index so the resulting state snapshot was visible at the end of the trajectory;
4. verified that the target value changed and that the mutation retained its same-record or cross-namespace classification; and
5. re-ran the pinned official evaluator.

All ten public tool calls succeeded. The intended target value was reached in all ten, the assigned class remained unchanged in all ten, and the official evaluator again returned 1.0 with an unchanged evaluation hash in all ten. Four reminder-tool cases produced two persistent field changes because the public tool automatically refreshed the creation timestamp; the other six produced one. This post-hoc result is not added to the confirmatory denominator, but it falsifies the reachability and backdating explanations.

## Code-level mechanism

The pinned ToolSandbox evaluator defines task success through milestone and minefield matchers.

- `MilestoneMatcher.compute_mapping_and_similarity` searches trajectory snapshots for the mapping that maximizes the declared milestone similarities.
- `EvaluationResult` sets the overall similarity to milestone similarity unless a minefield matches.
- None of the six affected task templates has a minefield.
- A milestone constraint evaluates only its declared namespace and target columns. The automatic guardrail builder removes a namespace when that milestone already constrains it.

These rules explain the two observed patterns:

- In same-record misses, the target namespace was constrained, but the added field was not among the milestone's target columns. Because that namespace was already constrained, no whole-namespace guardrail rejected the extra field.
- In cross-namespace misses, the required milestones could be matched before the appended public side-effect snapshot. The scorer does not require its final matched milestone to cover the final persistent state, so the trailing state change was outside the best mapping.

The narrow conclusion is that these six pinned task evaluators lack a final-state completeness condition for the tested reachable trailing effects. This is not a claim that all ToolSandbox scenarios or later releases share the same behavior.

## Evidence files

- Frozen misses: `analysis/tables/table6_false_negatives.csv`
- Diagnostic records: `analysis/toolsandbox_public_tool_diagnostic.jsonl`
- Diagnostic aggregate: `analysis/toolsandbox_public_tool_diagnostic_summary.json`
- Diagnostic code: `analysis/diagnose_toolsandbox_public_effects.py`
- Pinned evaluator source: `C:/Users/Sun/Documents/res/third_party/ToolSandbox-main/tool_sandbox/common/evaluation.py`

API calls: 0. Paid cost: CNY 0.
