# Hidden-Requirements Benchmark v2: Additional Experiments

This package extends the 100-task Communications AI & Computing benchmark with
the additional experiments recommended by the internal reviews.

## Why this v2 exists

The reviews identified three immediate experimental fixes:

1. The old `clarify_oracle` condition was not a true upper bound, because it
   revealed at most k=4 hidden requirements while some tasks have 5. In v2,
   the old condition should be relabeled as `budgeted_oracle`, and a new
   `full_oracle` condition reveals all hidden requirements.
2. The old visible checklist/self-check condition can contaminate strict output
   constraints such as JSON validity or exact bullet count. In v2,
   `clarify_verify_clean` performs the checklist internally and outputs only
   the final answer.
3. A new `clarify_no_info` control asks the same forced questions but receives
   no hidden information. This separates the effect of asking questions from the
   effect of actually revealing missing constraints.

## Critical implementation detail: temperature

`--temperature` defaults to `none`, and the API request omits the temperature
parameter unless you explicitly pass a numeric value. This avoids errors on
OpenAI, Anthropic, or Gemini endpoints that reject explicit temperature settings.

## New conditions

- `full_oracle`: reveals all hidden requirements, then asks for the final answer.
- `full_oracle_verify_clean`: reveals all hidden requirements and asks the model
  to verify internally, but output only the final answer.
- `clarify_verify_clean`: forced menu questions, simulator answers matching
  questions, internal verification only.
- `clarify_no_info`: forced menu questions, but all answers say there is no
  special hidden constraint. This is a control for prompt/interaction effects.
- `budgeted_oracle`: alias for the old oracle setup; reveals up to `clarify_k`
  requirements. Use this only if rerunning or reproducing old results.

## Recommended workflow

### 1. Smoke test

```bash
cd llm_hidden_requirements_benchmark_comm_ai_v2
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

export OPENAI_API_KEY="..."
export ANTHROPIC_API_KEY="..."
export GEMINI_API_KEY="..."

bash scripts/run_openai_smoke_new_conditions.sh
```

### 2. Prepare previous results

If you have the old raw CSV from the 100-task run:

```bash
python scripts/prepare_existing_results.py \
  --inputs /path/to/raw_results.csv \
  --out analysis_v2/existing_relabelled.csv
```

If you still have old JSONL result files:

```bash
python scripts/prepare_existing_results.py \
  --inputs "old_results/*.jsonl" \
  --out analysis_v2/existing_relabelled.csv
```

This relabels:

- `clarify_oracle` -> `budgeted_oracle`
- `clarify_verify` -> `clarify_verify_visible`

### 3. Run additional experiments

```bash
bash scripts/run_additional_experiments.sh
```

By default this runs:

- OpenAI `gpt-5.4`
- Anthropic `claude-opus-4-7`
- Gemini `gemini-3.1-pro-preview`

on:

- `full_oracle`
- `clarify_verify_clean`
- `clarify_no_info`
- `full_oracle_verify_clean`

### 4. Aggregate old + new results

```bash
python scripts/aggregate_results.py \
  --inputs "analysis_v2/existing_relabelled.csv" "results_extra/*.jsonl" \
  --outdir analysis_v2
```

### 5. Failure decomposition

```bash
python scripts/failure_decomposition.py \
  --inputs "analysis_v2/existing_relabelled.csv" "results_extra/*.jsonl" \
  --outdir analysis_v2
```

This produces paired contrasts such as:

- full information gain: `full_oracle - one_shot`
- question-selection gap: `budgeted_oracle - clarify_forced`
- information-budget gap: `full_oracle - budgeted_oracle`
- clean verification gain: `clarify_verify_clean - clarify_forced`

### 6. Multiplicative baseline

```bash
python scripts/multiplicative_baseline.py \
  --inputs "analysis_v2/existing_relabelled.csv" "results_extra/*.jsonl" \
  --outdir analysis_v2
```

This estimates requirement-level success rates and predicts strict-pass rates
as the product of per-requirement success probabilities.

## Output files

After aggregation and decomposition, the most important files are:

- `analysis_v2/summary_by_condition.csv`
- `analysis_v2/summary_by_model_condition.csv`
- `analysis_v2/summary_by_requirement_count.csv`
- `analysis_v2/requirement_level_summary.csv`
- `analysis_v2/failure_decomposition_contrasts.csv`
- `analysis_v2/revealed_vs_unrevealed_success.csv`
- `analysis_v2/multiplicative_baseline_summary.csv`
- `analysis_v2/pass_rate_by_condition_dot_ci.png`
- `analysis_v2/pass_rate_by_requirement_count.png`
- `analysis_v2/requirement_level_success_heatmap.png`

## Interpretation targets

The revised paper can use these contrasts to separate three failure modes:

1. **Omitted-information failure**: one-shot/template fail because the required
   constraints are absent.
2. **Elicitation failure**: forced clarification asks the wrong questions or
   reveals too few constraints.
3. **Execution failure**: even full oracle does not reach 100%, meaning the model
   can still fail to execute known constraints.

This is the experimental structure needed before adding mixed-effects statistics
and rewriting the paper.
