# Research: Task-scoped side-effect calibration (SQJ-first)

## Venue Target

- **Primary:** Software Quality Journal (SQJ), subscription route.
- **Tier assumption:** public CCF 2026 directory mirror lists SQJ as C in
  software engineering; verify the relevant directory version with the
  institution before submission.
- **Cost constraint:** target 0 CNY submission/publication charge by declining
  optional Open Access. No model/API budget is required by this experiment.
- **Speed constraint:** first editorial decision is the planning milestone; it
  is not an acceptance guarantee.
- **Venue-fit amendment:** `venue-target-amendment-v1.2.md`.
- **Model-arm amendment:** `protocol-amendment-v1.3-model-arm.md`.
- **Operational/provider amendments:**
  `protocol-amendment-v1.6-model-arm-operational-freeze.md` and
  `protocol-amendment-v1.7-provider-substitution.md`.

## Goal

Establish, then exhaustively execute, a frozen task-scoped side-effect
calibration protocol that validates public stateful tool-agent evaluators as
software-quality artifacts. The confirmatory study must be reproducible on
Windows without model calls, paid APIs, model training, or private data. A
separate bounded model-trajectory arm may provide exploratory ecological
validity evidence, and the complete work must be written as an empirical
software-quality/evaluator-validation paper suitable for SQJ.

## Success Metric

- **Metric:** percentage of six preflight gates passed
- **Target:** 100/100, with every hard gate true
- **Direction:** maximize

The gates are: (1) pinned evaluator environments execute; (2) 20 eligible fixed
state tasks per benchmark execute; (3) gold acceptance is at least 95% per
benchmark; (4) a blinded manual audit accepts at least 90% of 30 generated
mutants as valid; (5) deterministic replay and append-only logging pass; and
(6) all records report zero model/API calls and zero paid cost. Gates 1--3 and
5 are the runtime preflight; gate 4 prevents the runtime preflight from being
misreported as the complete frozen-protocol preflight.

## Constraints

- **Max iterations:** 12
- **Time budget per experiment:** 5 minutes, except dependency/data installation
  commands whose integrity is checked separately
- **Pause for review every:** never
- **Evaluator:** `& '.\.venv\Scripts\python.exe' evaluate_preflight.py`
- **Keep policy:** pass_only
- **Guard:** the confirmatory arm has no model/API calls; no API keys; no
  Docker; no protected AppWorld material copied into public artifacts; no
  formal outcomes before preflight passes; frozen scientific metrics and
  mutation classes remain unchanged; no post-hoc venue-driven scientific
  changes. The secondary model arm must obey amendment v1.3 caps and cannot
  alter confirmatory conclusions.
- **Noise runs:** 1
- **Min delta:** 0

## Current Approach

Build independent adapters for the pinned tau2, ToolSandbox, and AppWorld
evaluators. Feed them benchmark-provided fixed/gold states rather than model
trajectories, and write append-only JSONL records with source, input, and output
hashes.

## Search Space

- **Allowed changes:** dependency isolation, adapter code, fixture extraction,
  deterministic serialization, logging, retry-safe resume, Windows path fixes,
  evaluator invocation mechanics, and the bounded exploratory model arm in
  amendment v1.3.
- **Forbidden changes:** changing confirmatory benchmark inclusion rules,
  mutation classes, success metrics, statistical methods, or calling a model
  to manufacture a passing state. The model arm may not be promoted to the
  confirmatory arm after results are observed.

## Context & References

- Frozen protocol: `..\topic-search-2026-08-01\frozen-protocol-v1.md`
- Novelty audit: `..\topic-search-2026-08-01\novelty-audit-go.md`
- tau2 pin: `fc0055dc4e0a316c3f83133267fbd6faaa770992`
- ToolSandbox source hashes: `..\topic-search-2026-08-01\scope-feasibility-toolsandbox.json`
- AppWorld package: `0.1.3.post1`

## History

| # | Change | Metric | Result | Timestamp |
|---|---|---:|---|---|
| 0 | Frozen preflight scaffold | 0 | baseline; no gate measured yet | 2026-08-01T12:40:00+08:00 |
| 1 | Isolated environments and three official evaluator adapters | 66.67 | four of six gates passed; ToolSandbox replay exposed wall-clock hash drift | 2026-08-01T13:22:00+08:00 |
| 2 | Normalize ToolSandbox fixture clock origin and retest across processes | 83.33 | five of six gates passed; runtime preflight passed, blinded mutant audit pending | 2026-08-01T13:25:00+08:00 |
| 3 | SQJ-first venue-target amendment | pending | venue/framing changed; confirmatory science unchanged; blind audit still pending | 2026-08-06T+08:00 |
| 4 | Add bounded model-trajectory realism arm | pending | exploratory only; 36 episodes maximum; deterministic main arm unchanged | 2026-08-08T+08:00 |
| 5 | Harden one-command blind-audit workflow | 83.33 | frozen packet integrity passed; scoring now updates status atomically; human audit remains 0/30 | 2026-08-08T+08:00 |
| 6 | Complete blind audit and two-stratum runner smoke | 100.00 | 30/30 blind labels valid; all six gates and cross-process append-only replay passed; zero API/cost | 2026-08-08T14:03:00+08:00 |
| 7 | Add stratified generator-audit quality gate | 100.00 preflight / 60.00 formal readiness | original preflight remains passed; 15-item supplemental packet frozen before formal outcomes | 2026-08-08T14:31:00+08:00 |
| 8 | Correct S004 positional-list bug | 100.00 preflight / 66.67 formal readiness | invalid retained; primary-key diff tests and versioned three-benchmark replay pass; one replacement audit pending | 2026-08-08T14:46:00+08:00 |
| 9 | Replacement audit and frozen 20-task nine-stratum confirmatory run | 100.00 preflight / 100.00 confirmatory readiness | 15/15 effective stratified audit valid; 540 confirmatory cells validated with stable fingerprints and zero API/cost; two tau2 tail timeouts retained out of scope | 2026-08-08T20:20:00+08:00 |
| 10 | Pre-registered statistics, public-tool diagnosis, paper sections, and adversarial review | 100.00 deliverable verification | 145 applicable mutations analyzed; 10/10 ToolSandbox misses reproduced through public tools; seven tables, three figures, four manuscript sections, novelty review, and all consistency gates pass; zero API/cost | 2026-08-09T12:46:03+08:00 |
| 11 | Pre-outcome model-provider resource substitution | pending | Kimi replaced one-for-one by gateway-served `glm-5.2`; DeepSeek unchanged; new v1.7 study/log namespace; no live model-arm output observed | 2026-08-09T+08:00 |
| 12 | Correct gateway identity before live generation | pending | v1.7 AGICamp identification error retained and excluded; correct Token Rhythm endpoint frozen in v1.8; no live episode or paid generation occurred | 2026-08-09T+08:00 |
| 13 | Reorder pending providers and repair ToolSandbox client | pending | one completed Prism success and one pre-generation DeepSeek infrastructure error retained; remaining Qwen/Token first, DeepSeek next-to-last, Prism last; ToolSandbox OpenAI/httpx mismatch repaired | 2026-08-09T+08:00 |
| 14 | First QwenPlus and Token Rhythm live compatibility cells | 100.00 preflight / 3 of 36 model cells complete | Token Rhythm metadata probe passed for gateway-visible `glm-5.2`; both new AppWorld trajectories exhausted the frozen 12-call limit without completing the task or changing persistent state; failures retained as valid negatives; confirmatory conclusions unchanged | 2026-08-09T17:54:06+08:00 |
| 15 | Terminate low-value model arm and complete SQJ submission package | 100.00 submission verification | model arm retained at 7/36 and excluded below its inclusion gate; complete 5,000-word LaTeX draft, 15 verified references, allowed-delta implementation, eight tables, five 600-dpi figures, nine tests, and one-command key-free replication pass; zero new API/cost | 2026-08-09T23:40:00+08:00 |
| 16 | Freeze SJTU self-deployed-model ecological-validity supplement | 100.00 submission verification / supplement pending | Shanghai Jiao Tong University “Zhiyuan No.1” self-deployed model service on Huawei Ascend 910B NPU added as a separate six-episode descriptive arm; no confirmatory scope or manuscript primary claim changed; metadata-only probe and hidden-key wrapper added; zero API/cost | 2026-08-10T+08:00 |
