# Data dictionary

One line per column. Two files carry species labels in two forms; joining on
the wrong one silently drops rows, so the distinction is stated first.

## Species keys

Which key to use depends on what is being joined, and getting it wrong drops
rows without error.

- `species` — the original CORAL label (`Genus_species` as published), carrying
  strain and subspecies suffixes where the source had them:
  `Agaricus_bisporus_var._bisporus_H97`. **This is the key for joining the
  derived tables to each other.**
- `binom` — the reduced binomial, present in `confirmatory_aligned.csv` only:
  `Agaricus_bisporus`. It differs from `species` for 254 of the 4,972
  confirmatory entries. **This is the key for matching a species to a tip of
  the calibrated tree**, since the tree is labelled by binomial. Joining the
  tree on `species` instead silently retains 4,718 of 4,972, and the 254 lost
  are disproportionately fungi. `code/phylo_diagnostics.py` asserts the count
  for that reason.

## cohortB_R_per_species.csv — 5,053 rows

- `species` — CORAL label
- `R` — spectrum-conditioned advantage, the confirmatory estimand
- `z` — standardized separation from the randomized ensemble
- `p_emp` — empirical p-value: (codes with cost no greater than the standard code + 1) / (10,000 + 1)
- `Phi_SGC` — expected substitution cost under the standard code
- `Phi_null` — mean cost across the 10,000 randomized codes
- `sd_null` — standard deviation of those costs
- `sigma2` — measurement variance of `R` from the multinomial bootstrap

## confirmatory_aligned.csv — 4,972 rows

The subset that resolves on the calibrated tree. Columns as above, plus `binom`.

## spectrum_components.csv — 5,053 rows

- `species`, `kappa_log` — natural log of the transition/transversion ratio
- `cpg_shift_log` — natural log of the CpG>T mutational shift

## R_all_metrics.csv — 5,053 rows

`R_polar`, `R_grantham`, `R_kd`: the estimand recomputed under each
amino-acid cost metric.

## cross_metric.csv — 3 rows

Mean, standard deviation, range and the regression coefficient on standardized
log kappa for each metric, over the 4,972 confirmatory species.

## per_clade_slopes_phylo.csv, cv_z_by_clade.csv, rmse_decomposition.csv, inverted_loco.csv

Per-clade fits. `train_clade` is the clade used for fitting, `gain` the
reduction in prediction error on the remainder.

## nonsense_share.csv — 4,972 rows

`nonsense`: share of outgoing mutational weight terminating on a stop codon.

## ensemble_summaries.npz

Per-code summaries over the 10,000 randomized codes: `slope`, `mean_R`,
`var_R`, and the corresponding standard-code values with the `_sgc` suffix.

## confirmatory_ensemble.json

The frozen ensemble: `n`, `seed`, `codon_order`, and `codes` as 64-character
strings. SHA-256 recorded in the manuscript and checked by
`code/verify_package.py`.

## coral_counts.pkl, coral_abund.pkl

Substitution counts and callable trinucleotide abundances aggregated per
species from the CORAL branch table, 5,053 x 96. These are derived aggregates,
not the source table; the source is third-party and is not redistributed here.

## timetree_5053_species.nwk

Calibrated phylogeny from TimeTree 5 for the 5,053 binomials; 44 exclusions
listed in `tree_excluded_species.txt`.

## phylo_diagnostics.json

Written by `code/phylo_diagnostics.py` over all 4,972 confirmatory species.

- `lambda_hat`, `tau2_hat` — maximum-likelihood estimates
- `lambda_ci`, `tau2_ci` — 95% profile intervals, located by interpolating the
  crossings of 2 delta log-likelihood = 3.84
- `recovery` — one entry per generating lambda, with the median and range
  recovered over six replicates and the ratio of recovered to generating tau2
- `order_lambda_range`, `order_tau2_cv` — lambda and the coefficient of
  variation in tau2 across six random polytomy resolutions
- `n_species` — 4,972, asserted

## optim_convergence.json

Written by `code/optim_convergence.py`.

- `budget_sweep` — one entry per (kappa, restarts, iterations): best and median
  cost found, the spread across restarts, the similarity to the standard code at
  the best solution and at the median, and how many restarts reach within 0.1%
  of the best
- `gain_kappa_N` — percentage improvement in best cost from the smallest to the
  largest budget at that kappa

## external_replication.json

Written from the independently inferred spectra. Counts of randomized codes
beating the standard code, the correlations of R, z and CV with log kappa, and
the tercile contrasts, over the species listed in `replication_species.csv`.

## external_source_manifest.csv — 113 rows

Every species in the third-party deposit, in the deposit's own alphabetical
order, with whether it was recovered and whether it was analysed. The two are
identical by construction: nothing recovered was set aside.

- `alphabetical_position` — 1 to 113, the deposit's own ordering
- `species` — species name as the deposit labels it
- `file` — the posterior-summary filename
- `recovered` — yes/no
- `analysed` — yes/no; equals `recovered` for every row

## external_replication_per_species.csv — 68 rows

Written by `code/replication.py`. One row per species of the independently
inferred set, and the table from which every figure of the external replication
is recomputed by the audit.

- `species` — species name as deposited by the source
- `kappa` — transition/transversion ratio of that species' spectrum
- `R` — advantage of the standard code relative to the ensemble mean
- `z` — standardized separation, R / CV
- `CV` — coefficient of variation of the randomized-code costs
- `rank` — randomized codes of strictly lower cost, out of 10,000
- `p_emp` — conservative empirical p-value, using cost no greater than

## replication_species.csv

One column, `species`: the species used for the external replication, in
alphabetical order. Written by `code/replication.py` from the per-species table
on every run.

## divergence.json

Written by `code/divergence.py`. The rank range and median, the percentage of
species with no better code, the decile contrasts in R, z, CV and rank, and the
correlations of each with log kappa.

## coral_name_mapping.json

Maps the CORAL label to the binomial used against the tree, for the 254 entries
where they differ.

## timetree_species_list.txt

The tip labels of the calibrated tree, one per line, as supplied.
