# Hidden-Requirements Benchmark for Communications AI & Computing Submission

This package runs a 100-task hidden-requirements benchmark for testing one-shot
objective specification, structured prompting, and clarification protocols.

## Key design note

`--temperature` defaults to `none`, so the runner omits the temperature parameter
from OpenAI, Anthropic, and Gemini API calls unless you explicitly set it. This is
important because some current model endpoints reject user-specified temperature.

## Conditions

- `one_shot`: raw user request only.
- `template`: structured one-shot prompt with a general checklist.
- `clarify_forced`: model must ask up to `k` clarification questions from a fixed menu; then it answers.
- `clarify_verify`: same as forced clarification, but the answer phase requires a checklist and self-check.
- `clarify_oracle`: benchmark reveals up to `k` true hidden requirements; this is an upper-bound protocol.

## Recommended main experiment

This is the suggested Communications AI & Computing-scale experiment:

- 100 tasks
- 3 strong models
- 5 conditions
- `clarify_k=4`
- `temperature=none`
- optional 3-repeat subset for robustness

This creates roughly 100 x 3 x 5 benchmark rows, but more API calls because
forced/verify clarification use two calls per task.

## Setup

```bash
cd llm_hidden_requirements_benchmark_comm_ai
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

export OPENAI_API_KEY="..."
export ANTHROPIC_API_KEY="..."
export GEMINI_API_KEY="..."
```

## Smoke test

```bash
python scripts/run_benchmark.py \
  --provider openai \
  --model gpt-5.4 \
  --condition template \
  --limit 3 \
  --out results/smoke_openai_template.jsonl
```

## Full strong-model suite

```bash
bash scripts/run_comm_ai_suite.sh
```

## Aggregate results

```bash
python scripts/aggregate_results.py \
  --inputs "results/*.jsonl" \
  --outdir analysis
```

## Human audit export

Export a stratified sample of outputs for manual validation of the automatic grader:

```bash
python scripts/export_human_audit.py \
  --inputs "results/*.jsonl" \
  --out analysis/human_audit_sample.csv \
  --n 120
```

After manually filling `human_overall_pass`, run:

```bash
python scripts/merge_human_audit.py \
  --audit analysis/human_audit_sample.csv \
  --out analysis/human_audit_agreement.csv
```

## Reproducibility

The file `data/tasks.jsonl` is generated deterministically from `scripts/generate_tasks.py`
with seed 20260425. You can regenerate it with:

```bash
python scripts/generate_tasks.py --n 100 --seed 20260425 --out data/tasks.jsonl
```
