# Online Resource 1 — Supplementary Materials

**Paper:** *How Well Do Existing Agentic AI Taxonomies Characterize Real-World Incidents? An Empirical Validation of the OWASP ASI, Microsoft AI Red Team, and NIST AI RMF Frameworks Using Public Incident Data*

**Author:** Withheld for double-anonymous review

**Target journal:** Springer *AI and Ethics*

**Submission date:** July 20, 2026

---

## Purpose

This archive contains all materials required to independently replicate, verify, or extend the empirical analysis reported in the paper. It is referenced eight times in the paper text (in §3.1, §3.3, §3.4, §3.5, §9, and Appendix A). Reviewers, editors, and downstream researchers should find here everything needed to reproduce every reported number and every figure.

---

## Directory structure

```
Paper3_Supplementary/
├── README.md                              this file
├── code/                                  four Python scripts, run in order
│   ├── 01_aiid_retrieval.py               pull all AIID incidents via GraphQL
│   ├── 02_classify_agent_related.py       apply weighted keyword filter (Appx. A)
│   ├── 03_analysis_and_stats.py           coverage, per-category, temporal splits
│   └── 04_kappa_computation.py            Cohen's kappa on Pass 1 vs Pass 2
├── data/                                  primary + derived data
│   ├── agent_related_incidents_n131.json  the coded sample (N = 131)
│   ├── non_agent_incidents_n1422.json     Stage A failures (audit trail)
│   ├── coded_workbook.xlsx                full coding sheet with all three frameworks
│   ├── coded_workbook.csv                 same, plain-text mirror
│   ├── owasp_codes.csv                    OWASP-ASI reference sheet
│   ├── microsoft_codes.csv                MSFT-AIRT reference sheet
│   ├── nist_functions.csv                 NIST AI RMF reference sheet
│   ├── analysis_summary.json              computed coverage / frequency stats
│   └── kappa_results.json                 reliability statistics
├── codebook/
│   └── codebook.md                        boundary-decision log (§3.3, updated per §6)
└── figures/                               all seven figures at 200 DPI
    ├── fig1_frameworks.png                schematic overview of three frameworks
    ├── fig2_owasp_freq.png                OWASP category frequency
    ├── fig3_msft_freq.png                 Microsoft top-15 modes
    ├── fig4_nist.png                      NIST function distribution
    ├── fig5_coverage_temporal.png         coverage by pre-2022 / 2022+ subset
    ├── fig6_temporal.png                  year distribution of coded sample
    └── fig7_cooccurrence.png              OWASP × Microsoft co-occurrence heatmap
```

---

## Quick verification path (15 minutes)

For a reviewer confirming the reported numbers without re-running data retrieval:

1. Open `data/coded_workbook.xlsx` (or `.csv`) — this is the source of truth.
2. Open `data/analysis_summary.json` — compare against §4 numbers in paper.
3. Open `data/kappa_results.json` — compare against §4.1 kappa values.
4. Open `codebook/codebook.md` — review boundary decisions and reconciliation log.

Every number reported in the paper's §4 through §6 flows from these four files.

---

## Full replication path (2 hours)

For a reviewer or replicating researcher who wants to reproduce the full analysis from the AIID source:

**Prerequisites:**
- Python 3.11 or newer
- `pip install requests pandas scikit-learn openpyxl matplotlib`
- Internet access (AIID GraphQL endpoint)

**Steps:**

```bash
# 1. Retrieve full AIID corpus (~2 minutes)
cd code/
python 01_aiid_retrieval.py
# → creates aiid_output/agent_related_incidents.json
# → creates aiid_output/non_agent_incidents.json

# 2. Apply weighted keyword filter (~10 seconds)
python 02_classify_agent_related.py
# → creates agent_related_incidents_n131.json (Stage A pass)
# → creates non_agent_incidents_n1422.json    (Stage A fail)

# 3. Compute coverage, frequency, temporal-subset statistics (~5 seconds)
# Requires the coded workbook (data/coded_workbook.xlsx) to be present
python 03_analysis_and_stats.py

# 4. Compute Cohen's kappa on Pass 1 vs Pass 2 (~2 seconds)
python 04_kappa_computation.py
```

The retrieval step (01) yields a fresh AIID corpus; incidents added to AIID after July 3, 2026 will be included but the Stage B human review step (§3.2 of paper) has not been performed on them. Reviewers reproducing the sample should either restrict the retrieval date range to on-or-before July 3, 2026 or accept a superset of the reported sample.

---

## Deviations from best-practice, disclosed

Three deviations from full best-practice methodology are documented in the paper and repeated here for reviewer transparency.

**1. Single-coder design (§3.4, §9).** All 131 incidents were coded by a single coder. This is disclosed in the paper as the primary limitation and future-work extension.

**2. AI-assisted coding procedure (§3.3.2).** A large language model (Anthropic Claude) was used to generate initial coding suggestions. The author independently reviewed, accepted, modified, or rejected each proposed assignment before finalizing the dataset. Final coding decisions rest with the author. The intra-rater reliability estimates therefore measure the stability of this human-in-the-loop procedure. Independent replication with purely human coders and with alternative LLM aids are both directions the released data was designed to support.

**3. Compressed Pass 1 – Pass 2 interval (§3.4).** The methodological ideal for intra-rater reliability testing is a temporal separation of at least two weeks between coding passes. Due to submission-cycle constraints, the Pass 2 subsample of 33 incidents was re-coded less than one day after Pass 1. The reported kappa figures (OWASP κ = 0.703; Microsoft κ = 1.000; NIST κ = 0.954) should therefore be interpreted as upper bounds on true intra-rater stability. This deviation is disclosed openly and the underlying coding data is provided so that any reviewer or future researcher can independently reproduce or re-perform the reliability calculation under a longer interval.

---

## Data provenance

**Primary source.** AI Incident Database (https://incidentdatabase.ai), maintained by the Responsible AI Collaborative under CC-BY-4.0. Accessed via public GraphQL endpoint on July 3, 2026. Corpus size at access: N = 1,553 incidents. Attribution: McGregor 2021.

**Secondary sources.** Vendor security disclosures from OpenAI, Anthropic, Google DeepMind, and Microsoft covering January 2024 through May 2026. Regulatory case profiles from the EU AI Act incident tracker and NIST AI RMF illustrative case studies. Prefixed VEN- and REG- respectively in incident identifiers.

**Frameworks coded against.**
- OWASP Top 10 for Agentic Applications 2026 (release date: December 2025)
- Microsoft AI Red Team Taxonomy of Failure Modes in Agentic AI Systems (release: April 2025)
- NIST AI Risk Management Framework 1.0 (release: January 2023) + Generative AI Profile (release: July 2024)

---

## Licensing

The AIID source data is CC-BY-4.0 licensed; this supplementary archive preserves that license for all AIID-derived content. The coded workbook, codebook, and analysis scripts are released by the author under CC-BY-4.0 for consistency and for maximum reusability by the research community.

---

## Contact

For questions about replication or extension, or to report an error in the coded data, contact the corresponding author through the Springer *AI and Ethics* editorial office.

---

## Version

Version: 1.0
Date: July 3, 2026
