The authors' check (consolidated sheet data/author_check.csv: 126 items = all 76 items on which the two passes disagree + 50 items drawn at random from the 601 on which they agree)
The sheet has no notes. Each author checked the items separately and the judgments were consolidated into one code per item; only the consolidated sheet is on file, so the agreement below is between the authors' consolidated judgment and the machine coding, not between the authors. The sheet showed the type of check and the codes of the two passes, so the check is not independent of them.

Distribution of the authors' codes: Y 5, E 43, N 60, X 18

1. The 76 items on which the passes disagree: which side the authors take
  same as pass 1: 23; same as pass 2: 50; different from both: 3
  Cross-tabulation (rows = pass 1/pass 2, columns = authors):
author   E   N  X  Y
row_0               
E/N      4  17  0  1
E/Y      5   1  0  1
N/E     13   9  1  0
N/X      0   1  6  0
X/E      0   0  1  0
X/N      0   1  2  0
Y/E     10   0  0  0
Y/N      0   2  0  1

2. The 50 random items (passes agree): agreement of the authors with the shared code
  four-category agreement 45/50; related/unrelated agreement 47/50; X vs not-X agreement 49/50
  Cross-tabulation (rows = shared code, columns = authors):
author    E   N  X  Y
p1_code              
E        10   3  0  0
N         0  25  0  0
X         0   1  8  0
Y         1   0  0  2

3. Estimated for all 677 new items (disagreement stratum complete, agreement stratum weighted 601/50): agreement and kappa between the authors and the machine passes
  The agreement is a stratified estimate and kappa is computed from the weighted cross-tabulation; the intervals are 95% percentile intervals of a stratified bootstrap (2,000 resamples, seed 20260935).
  This is agreement between a human judgment and a model, not reliability between the authors; the authors' judgment is not a gold standard, only a check.
  authors vs pass 1 | four-category: agreement 83.3% [75.3%, 90.2%], kappa = 0.735 [0.603, 0.842]
  authors vs pass 1 | related (Y/E) vs unrelated: agreement 89.8% [83.1%, 95.0%], kappa = 0.764 [0.601, 0.884]
  authors vs pass 1 | Y vs not-Y: agreement 96.2% [92.3%, 98.7%], kappa = 0.640 [-0.003, 0.880]
  authors vs pass 2 | four-category: agreement 87.3% [79.3%, 94.1%], kappa = 0.798 [0.666, 0.905]
  authors vs pass 2 | related (Y/E) vs unrelated: agreement 92.0% [85.4%, 97.2%], kappa = 0.815 [0.661, 0.929]
  authors vs pass 2 | Y vs not-Y: agreement 97.0% [93.0%, 99.4%], kappa = 0.700 [-0.005, 0.941]

  For comparison: between the two passes (677 new items) four-category kappa 0.817, related/unrelated kappa 0.833, Y/not-Y kappa 0.740 (ext_kappa_output.txt).

4. Agreement by group (descriptive; the group was not shown to the authors)
  labelled: 45 items checked; of these 33 disagreements, authors same as pass 1 8, same as pass 2 24; 12 random items, authors same as the shared code 10
  ordinary: 81 items checked; of these 43 disagreements, authors same as pass 1 15, same as pass 2 26; 38 random items, authors same as the shared code 35

5. Sensitivity analysis with the authors' codes substituted
  The authors' judgment replaces the 126 checked items (the other new items have the same code in both passes; the 90 original items keep their original blind codes). 58 items differ from the pass-1 code and 31 differ from the pass-2 code.
  Of the labelled pairs: different from pass 1 in 27/45
  Of the ordinary pairs: different from pass 1 in 31/81
  Direction of the changes (authors relative to pass 1; the 126 checked items only, descriptive, the group was not shown):
    labelled pairs (45 items): unchanged 18, Y/E within related 12, related→unrelated 10, unrelated→related 5
    ordinary pairs (81 items): unchanged 50, related→unrelated 13, unrelated→related 8, non-X→X 7, X→non-X 2, Y/E within related 1
  Of the 76 disagreements, in the 73 where the authors agree with one of the passes they side with pass 2 in 50 (pass 1 in 23); symmetric two-sided binomial test p = 0.0021 (descriptive).


  identity check (no substitution, pass-1 codes): identical, cell by cell, to the corresponding rows of yisheng_models_ext_coding.csv

  test                                   | pass 1 (control)                       | authors' judgment substituted
  P1 near ~ label | related
      control:      73/141 vs 23/79, OR 2.35 [1.43, 3.86], p1 0.00038; MH 2.01 [0.75, 5.39]
      substituted:  70/138 vs 19/74, OR 2.73 [1.61, 4.65], p1 0.0001; MH 2.26 [0.80, 6.33]
      rule:         supports C (same word / minimal derivation)
  S1 MC identical ~ label | related
      control:      54/141 vs 15/79, OR 2.57 [1.35, 4.87], p1 0.0019; MH 1.57 [0.57, 4.32]
      substituted:  50/138 vs 12/74, OR 2.93 [1.48, 5.80], p1 0.001; MH 2.37 [0.78, 7.20]
      rule:         supports C (same word / minimal derivation)
  S2 near ~ label | related, paronomastic glosses removed
      control:      43/94 vs 20/73, OR 2.09 [1.18, 3.71], p1 0.0059; MH 1.36 [0.33, 5.66]
      substituted:  41/92 vs 16/68, OR 2.44 [1.32, 4.52], p1 0.0022; MH 1.21 [0.27, 5.34]
      rule:         supports C (same word / minimal derivation)
  S3 near ~ label | related, labels shared with Xiao Xu only
      control:      41/93 vs 23/79, OR 1.81 [1.04, 3.15], p1 0.017; MH 1.24 [0.38, 4.07]
      substituted:  39/91 vs 19/74, OR 2.08 [1.17, 3.71], p1 0.0064; MH 1.1 [0.29, 4.19]
      rule:         supports C (same word / minimal derivation)
  X3 near ~ label | related, shared and non-paronomastic (exploratory)
      control:      27/66 vs 20/73, OR 1.8 [0.95, 3.42], p1 0.036; MH 1.43 [0.29, 7.16]
      substituted:  26/65 vs 16/68, OR 2.14 [1.08, 4.25], p1 0.015; MH 1.24 [0.23, 6.71]
      rule:         supports C (same word / minimal derivation)
  S6 near ~ label | coded Y only
      control:      24/38 vs 0/4, not estimable
      substituted:  17/26 vs 0/5, not estimable
      rule:         not estimable (GEE; e.g. complete separation)
  X1 near ~ label | coded E only (exploratory)
      control:      49/103 vs 23/75, OR 1.8 [1.00, 3.25], p1 0.025; MH 0.84 [0.27, 2.63]
      substituted:  53/112 vs 19/69, OR 2.14 [1.19, 3.87], p1 0.0057; MH 1.25 [0.41, 3.84]
  S8 H2 related ~ label | all coded pairs, X excluded
      control:      141/172 vs 79/462, OR 21.57 [13.31, 34.98], p1 6.5e-36; MH 19.77 [10.10, 38.69]
      substituted:  138/172 vs 74/457, OR 20.74 [13.02, 33.05], p1 1.3e-37; MH 22.82 [11.33, 45.94]
  S9 near ~ related + label: coefficient of label
      control:      81/172 vs 83/462, OR 1.98 [1.30, 3.02], p1 0.00076
      substituted:  81/172 vs 82/457, OR 2.25 [1.44, 3.51], p1 0.00017
  S9 near ~ related + label: coefficient of related
      control:      96/220 vs 68/414, OR 2.34 [1.51, 3.63], p1 7.8e-05
      substituted:  89/212 vs 74/417, OR 1.91 [1.21, 3.00], p1 0.0027
  X4 near ~ label | related, new items only (exploratory)
      control:      59/116 vs 21/73, OR 2.42 [1.37, 4.25], p1 0.0011; MH 1.26 [0.41, 3.90]
      substituted:  56/113 vs 17/68, OR 2.83 [1.54, 5.20], p1 0.00041; MH 1.38 [0.44, 4.31]
