Bias sensitivity (tipping-point) analysis (exploratory; uses the existing pass-1 codes, no new coding)
For the model and its assumptions see the head of ext_bias_sensitivity.py. In short: if (A) is true the true OR is 1; coding error is unrelated to sound but may differ between labelled and ordinary pairs.
Key points (numbers in the sections above; exploratory, the primary test is unchanged):
  1. If (A) is true (true OR = 1), bringing the pooled OR 2.61 down to 1 requires that 11.5% of the truly unrelated ordinary pairs were coded "related", i.e. about 50 of the 79 ordinary pairs coded related (63%) are false positives; bringing it to 2 needs 5.9% (30%), to 1.5 needs 9.1% (48%); bringing the lower 95% bootstrap limit to 1 needs 5.8%. The labelled-group rate hardly affects these figures (for fp_L from 0 to 30%, OR* = 1 needs 11.5% to 11.9%); with equal rates in the two groups 11.6% suffices.
  2. So what is needed is not different error in the two groups but the base: 383 ordinary pairs against 31 labelled pairs are coded unrelated. If (A) is true, "true relatedness" is a confounder, and the coders' measurement error in it leaves residual confounding in the ordinary group; the ordinary-group false positives fall mainly among the majority of not-near pairs and lower the near share of the ordinary pairs coded related.
  3. For comparison, between the two passes the ordinary-group false-positive rate is 4.2% (of the 73 ordinary pairs coded related in pass 1, 19% are not confirmed by pass 2); corrected by it, OR* = 2.20 [1.18, 4.34]; random error does not wipe out the difference.
  4. The authors' check sample gives 10.2% [2.5%, 20.6%] (of the ordinary pairs coded related in pass 1 the authors judged 47% [14%, 89%] unrelated, of the labelled pairs 16% [3%, 39%]); corrected by it, OR* = 1.29 [0.00, 3.84] (10.2% is close to the critical value 11.5%), but the interval is too wide to settle the question. The reason is that the random stratum contains only 5 ordinary pairs coded related in pass 1 (2 judged unrelated by the authors), the others mostly coming from pairs on which the two passes disagreed; and 53 ordinary pairs coded related have no author judgment.
  5. Conclusion: random error between the two passes is not enough to wipe out the difference. How large a systematic bias shared by both passes (if any; most likely a threshold for E wider than the authors') would have to be: to lower OR* to 2 or the lower limit to 1 takes about 5.9%, to lower it to 1 about 11.5%; the present check sample can neither confirm nor exclude it. Having the 53 ordinary pairs coded related that have no author judgment judged as well would turn "what share are false positives" into a count.
  6. The other route (relatedness missed by the gloss-based coding among near ordinary pairs): a truly related near ordinary pair has only 0.38 ([0.21, 0.65]) of the chance of a not-near one of being coded related, i.e. about 37 of the 60 near ordinary pairs coded unrelated would in fact be related, while almost none of the not-near pairs are missed; the coders do not see the readings, so this would require links of meaning to hide outside the glosses more often in homophonous ordinary pairs than in labelled ones. Of these pairs the authors' check covered only 6, of which 1 were judged related; too few to say.


1. The observed table (primary-test sample: in the frame, with MC readings, pass-1 code not X; 634 pairs; the same as P1)
  Coded related: labelled near 73, not near 68; ordinary near 23, not near 56
  Coded unrelated: labelled near 8, not near 23; ordinary near 60, not near 323
  Pooled odds ratio 2.61 (GEE primary test 2.35 [1.43, 3.86]; the two differ only in weighting)
  Near share among pairs coded related: labelled 73/141 = 52%, ordinary 23/79 = 29%.
  Pairs coded unrelated: 383 ordinary pairs against only 31 labelled pairs, 12 times as many: far more truly unrelated ordinary pairs are available to be miscoded as "related".

2. Tipping points: the false-positive rate fp_O in the ordinary group (false positive = truly unrelated but coded related) needed to bring the corrected OR* down to the target; fp_L (labelled group) takes the values below
  Upper limit of the ordinary-group false-positive rate (above it the 56 ordinary pairs coded related and not near would not suffice to be subtracted): 14.8%
  target | fp_L | fp_O needed | ordinary related pairs subtracted, as a share of the ordinary pairs coded related | true related ordinary pairs left
  OR* = 2 | 0% | 5.9% | 24/79 = 30% | 55
  OR* = 2 | 5% | 6.0% | 25/79 = 31% | 54
  OR* = 2 | 10% | 6.2% | 25/79 = 32% | 54
  OR* = 2 | 20% | 6.7% | 28/79 = 35% | 51
  OR* = 2 | 30% | 7.3% | 30/79 = 38% | 49
  OR* = 1.5 | 0% | 9.1% | 38/79 = 48% | 41
  OR* = 1.5 | 5% | 9.2% | 39/79 = 49% | 40
  OR* = 1.5 | 10% | 9.3% | 39/79 = 50% | 40
  OR* = 1.5 | 20% | 9.5% | 40/79 = 51% | 39
  OR* = 1.5 | 30% | 9.9% | 42/79 = 53% | 37
  OR* = 1 | 0% | 11.5% | 50/79 = 63% | 29
  OR* = 1 | 5% | 11.5% | 50/79 = 63% | 29
  OR* = 1 | 10% | 11.6% | 50/79 = 63% | 29
  OR* = 1 | 20% | 11.7% | 51/79 = 64% | 28
  OR* = 1 | 30% | 11.9% | 52/79 = 65% | 27
  lower 95% bootstrap limit of OR* = 1 | 0% | 5.8% | 24/79 = 30% | 55
  lower 95% bootstrap limit of OR* = 1 | 5% | 5.9% | 24/79 = 30% | 55
  lower 95% bootstrap limit of OR* = 1 | 10% | 6.0% | 24/79 = 31% | 55
  lower 95% bootstrap limit of OR* = 1 | 20% | 6.2% | 25/79 = 32% | 54
  lower 95% bootstrap limit of OR* = 1 | 30% | 6.5% | 27/79 = 34% | 52
  Equal false-positive rates (fp_L = fp_O): OR* = 1 needs 11.6%; bringing the lower limit of the bootstrap 95% interval down to 1 needs 5.9%
    Then only about 4/141 labelled pairs coded related are false positives (3%), against about 50/79 ordinary pairs (64%).
  On the GEE scale (matching the pooled 2.61 to 2.35, target OR* = 2.61/2.35 = 1.112): with fp_L = 0, fp_O = 11.0%; with fp_L = fp_O, 11.1% (against 11.5% and 11.6% on the pooled scale: less than one percentage point apart)

  How to read this: what is needed is not a difference between the false-positive rates of the two groups. Equal rates (about 11.6%) suffice, because there are many truly unrelated pairs in the ordinary group (383 pairs, against 31 labelled),
  so that with the same rate the share of false positives among the pairs coded related is much higher in the ordinary group (about 64%) than in the labelled group (about 3%). Put differently: if (A) is true, the near share among truly related ordinary pairs
  should equal that of the labelled group (73/68 = 1.07 to 1), whereas 23/56 = 0.41 is observed; to raise 23/56 to 1.07,
  about 50 of the 79 ordinary pairs coded related would have to be false positives (false positives do not choose by sound and fall mainly among the majority of non-near pairs).
  OR* falls with fp_O, ever more steeply (fp_L = 0; fp_O -> OR*): 0.0% -> 2.61; 2.5% -> 2.39; 5.0% -> 2.11; 7.5% -> 1.76; 10.0% -> 1.32; 11.5% -> 0.99

3. Comparison: the coding disagreements actually observed (all are disagreements between coders, not judgments relative to Xu Shen; see the limitations at the end)
  Pass 1 vs pass 2 (ordinary pairs, the 409 new items in the frame that are not X in pass 2):
    With pass 2 as reference: of the 333 pairs coded unrelated in pass 2, 14 were coded related in pass 1, a false-positive rate of 4.2% [2.1%, 6.8%]; of the 73 pairs coded related in pass 1, 19% [10%, 30%] are not confirmed by pass 2
    With pass 1 as reference: of the 336 pairs coded unrelated in pass 1, 17 were coded related in pass 2, a false-positive rate of 5.1% [2.9%, 7.4%]
  Pass 1 vs pass 2 (labelled pairs, the 139 new items in the frame that are not X in pass 2):
    With pass 2 as reference: of the 26 pairs coded unrelated in pass 2, 9 were coded related in pass 1, a false-positive rate of 34.6% [17.2%, 52.9%]; of the 116 pairs coded related in pass 1, 8% [3%, 13%] are not confirmed by pass 2
    With pass 1 as reference: of the 23 pairs coded unrelated in pass 1, 6 were coded related in pass 2, a false-positive rate of 26.1% [9.1%, 44.0%]
  (The disagreement between the two passes is roughly the sum of each pass's own random errors, so the random false-positive rate of a single pass is about half the figures above; systematic errors, where both passes make the same mistake, cannot be seen.)
  Of the items in the authors' check, 99 are in the frame and not X for the authors (disagreement stratum 61, random stratum 38; each random item stands for 601/50 = 12.02 items).
    Checked ordinary pairs coded related in pass 1: random stratum 5 (judged unrelated by the authors: 2), disagreement stratum 15 (judged unrelated by the authors: 11)
    Checked labelled pairs coded related in pass 1: random stratum 8 (judged unrelated by the authors: 1), disagreement stratum 23 (judged unrelated by the authors: 7)
  Pass 1 vs the authors (ordinary pairs): of the pairs the authors judged unrelated, the share coded related in pass 1 (false-positive rate) is 10.2% [2.5%, 20.6%]; of the pairs coded related in pass 1, the share the authors judged unrelated is 47% [14%, 89%]
  Pass 1 vs the authors (labelled pairs): of the pairs coded related in pass 1, the share the authors judged unrelated is 16% [3%, 39%]; the false-positive rate of the labelled group (95.0%) is not usable: very few labelled pairs were judged unrelated by the authors, almost all of them among those coded related in pass 1, so the estimate lies outside the feasible range of the model (fp_L < 75%).
  Of the 79 ordinary pairs coded related in pass 1, 6 belong to the original 100 items (coded by the authors themselves); of the 73 new items, 20 were checked (including those the authors judged X), and 53 have no author judgment.

  Estimates of the ordinary-group false-positive rate under each reference (compare the tipping point of section 2: OR* = 1 needs fp_O 11.5%, i.e. 50/79 = 63% of the ordinary related pairs would be false positives):
    The two passes (pass 2 as reference): fp_O 4.2%, equivalent to 17/79 = 21% of the ordinary related pairs being false positives
    The authors' consolidated judgment: fp_O 10.2%, equivalent to 43/79 = 55%; counted directly, the share of the ordinary pairs coded related in pass 1 that the authors judged unrelated is 47% (about 37/79)

4. The corrected OR* at the false-positive rates actually observed (labelled-group rate 0, or equal to the ordinary-group rate; the intervals include the resampling of phonetics and of the reference sample)
  The two passes (pass 2 as reference), fp_O = 4.2%, ordinary group only corrected (fp_L = 0): OR* = 2.20, 95% interval [1.18, 4.34]
  The two passes (pass 2 as reference), fp_O = 4.2%, equal rates in the two groups: OR* = 2.23, 95% interval [1.19, 4.39]
  The authors' consolidated judgment, fp_O = 10.2%, ordinary group only corrected (fp_L = 0): OR* = 1.29, 95% interval [0.00, 3.84]
  The authors' consolidated judgment, fp_O = 10.2%, equal rates in the two groups: OR* = 1.32, 95% interval [0.00, 3.90]
  (A lower limit of 0 means that in that resample the false-positive rate exceeded the upper limit of about 14.8%, the ordinary pairs coded related and not near were all subtracted, and OR* tends to 0.)
  An alternative estimate that does not use the false-positive model (direct recoding): the checked items stand for the whole new-item frame with their design weights, the original 100 items keep their codes, and relatedness is taken from the authors' judgment: pooled OR = 7.32, 95% interval [1.51, 30.98].
    Weighted number of pairs the authors judged related: labelled near 48, not near 82; ordinary near 4, not near 50. The ordinary near cell has only 4 weighted pairs, and the random stratum contains only 1 ordinary near pair coded related in pass 1 (each standing for 12), so this estimate is very unstable and goes in the opposite direction to the false-positive correction above; neither can count as evidence.
    The unweighted replacement of the checked items alone is reported in yisheng_models_ext_author_check.csv (P1, author_adjudicated).

4b. The other route: relatedness missed by the gloss-based coding among near ordinary pairs (false negatives, only in the ordinary group)
  If (A) is true and the sensitivity of the labelled group does not vary with sound, the observed OR equals the ratio of the sensitivities for not-near and near ordinary pairs, so a truly related near ordinary pair has only 1/2.61 = 0.38 of the chance of a not-near one of being coded related (95% interval [0.21, 0.65], from the bootstrap interval of the pooled OR).
  Put differently: if the not-near ordinary related pairs are not missed, about 37 of the 60 ordinary near pairs coded unrelated (62%) would in fact be related, whereas almost all of the 323 not-near ordinary pairs coded unrelated would be unrelated. The coders do not see the readings; for this to hold, links of meaning would have to hide outside the glosses more often in homophonous ordinary pairs than in labelled ones (the underestimate discussed in the summary).
  The authors' judgment: of these 60 near ordinary pairs, 4 belong to the original 100 items (coded by the authors themselves); of the new items 7 were checked, of which the authors judged 1 related (out of 6 not judged X); 49 have no author judgment. The numbers are too small to say much.

Limitations:
  1. The comparisons use disagreements between coders (the two passes; the authors' consolidated judgment). They measure random error in the coding and the difference between the authors and the machine passes, not a systematic departure from Xu Shen's own judgment; a systematic difference shared by all coders between gloss-based "relatedness" and the relatedness Xu Shen saw would not show up here. The authors' consolidated judgment is one sheet consolidated from the two authors' separate checks, and only the consolidated sheet is on file. The sheet showed the type of check and the codes of the two passes; if the authors were therefore drawn towards them, the false-positive rate estimated here is too low (a bias towards (C)).
  2. The model assumes that coding error is unrelated to sound (the coders do not see the readings, which the procedure guarantees). But the labelled pairs coded unrelated lie mostly on the not-near side (near 8/81, not near 23/91, Fisher p = 0.010), which shows that gloss-based "relatedness" and sound are linked in any case (as (C) predicts, but under (A) true relatedness and closeness in sound can also be linked); the correction treats error as unrelated to sound and cannot tell the two apart.
  3. "Related" is binary; its strength is not graded (see X1 and X2 for the effect of strength). Very few labelled pairs were judged unrelated by the authors, so the labelled-group false-positive rate cannot be estimated; only fp_L = 0 and fp_L = fp_O are given.
  4. The random stratum of the authors' check has only 38 items, each standing for about 12; the share of ordinary pairs the authors judged unrelated (47%) is determined mainly by the disagreement stratum (pairs on which the two passes differed); of the 5 ordinary related pairs in the random stratum only 2 were judged unrelated, so the interval is wide. To narrow it, the authors could judge the remaining 53 ordinary pairs coded related that have no judgment.
  5. The observed pooled odds ratio 2.61 and the GEE estimate 2.35 differ in weighting; the tipping points differ little between the two scales (see section 2).
  6. The most likely shared bias is the threshold for E (connected through one step of extension): nearly all ordinary pairs coded related are E (75/79), and the rule is to choose N when unsure between E and N. Counting only E (X1: 1.8 [1.00, 3.25]) and controlling for Y/E (X2: 2.13 [1.26, 3.63]) are reported in the results table; the direction is unchanged, but Y/E is itself coded and cannot replace checking every pair.
  7. This is an exploratory sensitivity analysis. It does not change the primary test fixed before the analysis (P1 OR 2.35, which the pre-specified rule reads as support for C) and involves no new coding.
