# AGENTSPILL: Excessive Agency Exploitation Audit

Systematic empirical audit of LLM tool-calling exploitation across three Gemini tiers. Six exploit classes, 162 real trials, 51.2% success rate.

## Quick Summary

| Exploit Class | Success Rate |
|---|---|
| A01 — Tool Description Injection | 78% (21/27) |
| A02 — Cross-Tool Ambiguity | 59% (16/27) |
| A03 — Tool Chain w/o Confirmation | 78% (21/27) |
| A04 — Phantom Tool Substitution | 22% (6/27) |
| A05 — Authority Confusion | 44% (12/27) |
| A06 — Context-Induced Agency | 22% (6/27) |

## Reproduction

### Prerequisites

- Python 3.11+
- OpenAI-compatible LLM API endpoint
- pdflatex (for paper compilation)
- Headless Chrome (for figure rendering)

### Setup

```bash
export OPENAI_BASE_URL="http://your-endpoint:PORT/v1"
export OPENAI_API_KEY="your-api-key"
export OPENAI_MODEL="your-model-id"
```

### Run Experiments

```bash
cd src
python run_experiments.py
```

Writes 162 JSON transcripts to `results/transcripts/`. Each transcript is keyed by exploit, variant, tier, and seed.

### Analyze Results

```bash
python analyze.py
```

Prints per-exploit and per-tier breakdowns, generates `results/figdata.json`, and runs the fail-closed verification that exits non-zero if any number mismatches the raw transcripts.

### Generate Figures

```bash
python make_figures.py                   # matplotlib data charts
# Concept diagram (SVG to PDF):
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
  --headless --disable-gpu \
  --print-to-pdf=figures/fig_concept.pdf \
  --no-pdf-header-footer \
  file://$PWD/figures/fig_concept.html
```

### Compile Paper

```bash
cd paper
pdflatex Paper.tex && pdflatex Paper.tex
```

### Verify All Numbers

```bash
python src/analyze.py
# VERIFY PASSED: 82/162 OK
```

The verification script re-reads every raw transcript, independently computes all metrics, and cross-checks against `paper/tables/macros.tex`. Any mismatch exits non-zero.

## Project Structure

```
├── src/
│   ├── gateway_client.py      # OmniRoute-compatible API client with tool execution registry
│   ├── exploits.py             # 6 exploit classes × 3 variants = 18 attack vectors
│   ├── run_experiments.py      # Runs all 162 trials, saves JSON transcripts
│   ├── analyze.py              # Parses transcripts, computes metrics, fail-closed verification
│   └── make_figures.py         # matplotlib charts (per-exploit bar, per-tier heatmap)
├── figures/
│   ├── fig_concept.html        # Concept diagram (SVG, 1200×720)
│   ├── fig_concept.pdf
│   ├── fig_exploit_rates.pdf   # Per-exploit success bar chart
│   └── fig_tier.pdf            # Per-tier per-exploit heatmap
├── results/
│   ├── figdata.json            # Summary data consumed by make_figures.py
│   └── transcripts/            # 162 raw JSON transcripts
├── paper/
│   ├── Paper.tex               # IEEE paper (8 pages)
│   ├── Paper.pdf
│   └── tables/
│       └── macros.tex          # Single source of truth for all numeric values
└── data/
```

## Citation

If you use this work in your research, please cite:

```bibtex
@article{rashidi2025agentspill,
  title={{AGENTSPILL}: Measuring How {LLM} Tool Descriptions, Cross-Tool
         Ambiguity, and Action Chains Compose into Excessive Agency Exploits},
  author={Rashidi, Mohammadreza},
  journal={arXiv preprint},
  year={2026},
  note={https://github.com/rezaduty/agentspill}
}
```

## License

MIT