# Protocol amendment v1.3: bounded model-trajectory realism arm

Issued on 2026-08-08 before the blinded audit and before formal mutation
scoring. This amendment adds a strictly secondary exploratory arm. The frozen
deterministic evaluator study remains the confirmatory main experiment.

## Why this arm is added

The primary experiment tells us which evaluator counterfactuals are accepted
or rejected. A small model-trajectory arm checks ecological validity: whether
real tool-agent trajectories can land in the evaluator blind spots identified
by the deterministic arm. It is not intended to rank models or estimate
production failure rates.

## New exploratory research question

- **RQ5 (exploratory):** On a fixed, predeclared subset of public tasks, how
  often does an evaluator accept a model-generated final state that a frozen
  delta-complete reference evaluator rejects for an observed harmful side
  effect?

RQ5 is descriptive. It has no confirmatory hypothesis, no provider ranking,
and no permission to change the primary mutation classes or statistics after
seeing results.

## Sampling and execution

- Select a six-task paired core: two tasks from each benchmark, using the
  deterministic SHA-256 ordering already used by the preflight catalog.
- Run QwenPlus, official-channel Kimi, official-channel DeepSeek, Prism
  `gpt-5.5`, and Prism `claude-sonnet-5` on the same six-task core.
- Run an additional six tasks with QwenPlus only, for a total of 12 QwenPlus
  episodes and 6 episodes for each other condition.
- Use one frozen agent prompt and one fixed tool/turn budget per benchmark.
  At most one retry is allowed for a transient transport error; all errors,
  timeouts, empty responses, and incomplete trajectories remain in the log.
- The task prompt receives only the public task description, public tool
  definitions, and initial state. It never receives gold answers, hidden
  labels, evaluator source paths, or mutation keys.
- A task is selected before any model output is inspected. The six-task core
  is used for paired descriptive comparisons; the QwenPlus expansion is not
  used to claim cross-provider differences.

## Identity and gateway policy

- `requested_alias`, returned model field, endpoint, timestamp, and provider
  metadata are recorded for every request.
- Prism aliases are reported as gateway-served aliases, never as verified
  official model weights or official provider results.
- A missing or unavailable provider is reported as missing data. It is not
  replaced after results are seen.
- API keys are read only from environment variables or hidden input and are
  never written to logs, screenshots, artifacts, or the manuscript.

## Outcomes

For each completed trajectory record:

1. official evaluator score and acceptance;
2. global-exact, task-projected, and delta-complete reference scores;
3. whether official acceptance conflicts with delta-complete rejection;
4. mutation/side-effect classification when public state evidence supports it;
5. prompt hash, requested/returned model, endpoint, token counts, latency,
   retry count, error type, experiment timestamp, code version, and file hash.

Report per-condition counts and Wilson intervals. The arm is exploratory and
does not receive multiple-testing claims or causal interpretation. It cannot
show how often deployed agents fail in the real world.

## Hard resource caps

- QwenPlus: at most 12 episodes and 1.2 million total tokens.
- Official Kimi: at most 6 episodes and 0.3 million total tokens.
- Official DeepSeek: at most 6 episodes and 0.3 million total tokens.
- Prism `gpt-5.5`: at most 6 episodes and 0.2 million total tokens.
- Prism `claude-sonnet-5`: at most 6 episodes and 0.2 million total tokens.
- Total model-arm cap: 2.2 million tokens and 36 episodes before retries.
  Retries may not exceed one per episode or 2x the per-condition token cap.
- The main deterministic experiment remains 0 CNY. For the secondary arm,
  keep Kimi and DeepSeek spend below 10 CNY each unless a later written
  amendment explicitly raises a ceiling. Stop on either the token cap or the
  user's provider balance warning, whichever comes first.

## Interpretation and Go/No-Go

- **Go for manuscript inclusion:** at least 24/36 episodes complete, all
  conditions have their logs, and the primary deterministic experiment has
  already passed its preflight.
- **Exploratory-only:** fewer than 24 complete episodes, any provider is
  unavailable, or gateway metadata cannot be recorded.
- **No-go for causal/model claims:** any result that depends on treating a
  Prism alias as an official model, cherry-picking successful episodes, or
  using the model arm to replace the deterministic mutation experiment.

