Raters: pre-validation (264 cells), independent_validation (80 cells), independent_legal_validation (80 cells), legal_crosscheck (264 cells)

Pairwise reliability:
  pre-validation vs independent_validation  n= 80  exact=57.5%  adjacent=95.0%  kappa=0.43  wkappa=0.73  AC2=0.79  wPABAK=0.77
  pre-validation vs independent_legal_validation  n= 80  exact=53.8%  adjacent=91.2%  kappa=0.38  wkappa=0.67  AC2=0.73  wPABAK=0.71
  pre-validation vs legal_crosscheck  n=264  exact=48.1%  adjacent=95.1%  kappa=0.24  wkappa=0.56  AC2=0.80  wPABAK=0.73
  independent_validation vs independent_legal_validation  n= 80  exact=72.5%  adjacent=100.0%  kappa=0.58  wkappa=0.82  AC2=0.92  wPABAK=0.89
  independent_validation vs legal_crosscheck  n= 80  exact=72.5%  adjacent=100.0%  kappa=0.58  wkappa=0.79  AC2=0.92  wPABAK=0.89
  independent_legal_validation vs legal_crosscheck  n= 80  exact=70.0%  adjacent=100.0%  kappa=0.54  wkappa=0.76  AC2=0.91  wPABAK=0.88

Mean across 6 rater pairs: exact=62.4%  adjacent=96.9%  kappa=0.46  weighted-kappa=0.72  AC2=0.85  wPABAK=0.81

Bootstrap 95% CIs (weighted kappa) for independent-vs-frozen-reference pairs:
  pre-validation vs independent_validation            weighted-kappa 95% CI [0.61, 0.82]
  pre-validation vs independent_legal_validation      weighted-kappa 95% CI [0.53, 0.77]

Primary-versus-not agreement (the boundary the gap analysis turns on):
  pre-validation     vs independent_validation           n= 80  agreement=88.8%  binary kappa=0.68
  pre-validation     vs independent_legal_validation     n= 80  agreement=88.8%  binary kappa=0.70
  pre-validation     vs legal_crosscheck                 n=264  agreement=95.1%  binary kappa=0.73

Final post-adjudication descriptive agreement (NOT independent validation):
  final          vs independent_validation  n= 80  exact=65.0%  adjacent=95.0%  kappa=0.53  wkappa=0.76  AC2=0.81  wPABAK=0.80
  final              vs independent_validation           n= 80  agreement=96.2%  binary kappa=0.89
  final          vs independent_legal_validation  n= 80  exact=61.3%  adjacent=91.2%  kappa=0.48  wkappa=0.70  AC2=0.76  wPABAK=0.74
  final              vs independent_legal_validation     n= 80  agreement=96.2%  binary kappa=0.90

Rater-substitution sensitivity for every zero-primary requirement:
  (would any column stop being "narrow" if an independent rater's code replaced ours?)
    A9       STABLE (no dissenting primary)
    GV       STABLE (no dissenting primary)
    IA10     STABLE (no dissenting primary)
    IA5      FRAGILE: M2 x IA5 <- legal_crosscheck; M5 x IA5 <- legal_crosscheck

Leave-one-decisive-cell-out sensitivity:
  (for each column, would removing its single decisive primary code change the verdict?)
    A10      Moderate on ONE cell (M8); drop it and the column becomes Narrow.
    A11      Moderate on ONE cell (M9); drop it and the column becomes Narrow.
    A12      Moderate on ONE cell (M11); drop it and the column becomes Narrow.
    A13      Moderate on ONE cell (M9); drop it and the column becomes Narrow.
    A14      Moderate on ONE cell (M10); drop it and the column becomes Narrow.
    A15      Broad on 3 cells; dropping any single cell leaves it Moderate.
    IA6      Broad on 4 cells; dropping any single cell leaves it Broad.
    IA626    Moderate on ONE cell (M11); drop it and the column becomes Narrow.
    IA7      Moderate on ONE cell (M8); drop it and the column becomes Narrow.
    IA8      Moderate on ONE cell (M9); drop it and the column becomes Narrow.
    M2.10    Moderate on ONE cell (M12); drop it and the column becomes Narrow.
    M2.11    Moderate on ONE cell (M2); drop it and the column becomes Narrow.
    M2.3     Moderate on ONE cell (M1); drop it and the column becomes Narrow.
    M2.9     Moderate on ONE cell (M6); drop it and the column becomes Narrow.
    MAP      Moderate on ONE cell (M8); drop it and the column becomes Narrow.
    MG       Moderate on ONE cell (M11); drop it and the column becomes Narrow.

Note: report the pairwise table and the mean; flag any cell where raters disagree by more than one level for adjudication.
