# Python (scipy) Validation of PowerShell Statistical Results

Generated: 2026-07-31 by `tools/validate_stats.py`

Python environment: numpy 2.5.1, scipy 1.18.0

Data: retraction_watch_gitlab.csv (71448 rows)

Cohorts: machine=9161, comparison=56852, machine_with_lag=9161, comparison_with_lag=56852


## Tolerances

This table is generated from the same list of tolerances the comparison below
applies, so the documented rule and the enforced rule cannot diverge. A row in the
comparison table is marked YES only if the absolute difference is at or below the
tolerance shown here for that statistic.

| Statistic | Tolerance | Basis |
|---|---|---|
| Median lag, machine (days) | 0.5 | implementation printing precision |
| Median lag, comparison (days) | 0.5 | implementation printing precision |
| Mann-Whitney U | 0.5 | implementation printing precision |
| Mann-Whitney z | 0.01 | printed to two decimals by the primary implementation |
| Cliff's delta | 0.001 | printed to three or more decimals by the primary implementation |
| MAD machine (days) | 0.5 | implementation printing precision |
| MAD comparison (days) | 0.5 | implementation printing precision |
| MAD difference (days) | 0.5 | implementation printing precision |
| MAD diff CI lower (days) | 0.5 | implementation printing precision |
| MAD diff CI upper (days) | 0.5 | implementation printing precision |
| MAD ratio | 0.001 | printed to three or more decimals by the primary implementation |
| Welch t (Brown-Forsythe) | 0.01 | printed to two decimals by the primary implementation |
| Welch df | 1 | implementation printing precision |
| Paper-mill % machine | 0.01 | printed to two decimals by the primary implementation |
| Paper-mill CI lower, machine | 0.01 | printed to two decimals by the primary implementation |
| Paper-mill CI upper, machine | 0.01 | printed to two decimals by the primary implementation |
| Paper-mill % comparison | 0.01 | printed to two decimals by the primary implementation |
| Paper-mill CI lower, comparison | 0.01 | printed to two decimals by the primary implementation |
| Paper-mill CI upper, comparison | 0.01 | printed to two decimals by the primary implementation |
| Paper-mill diff (pp) | 0.01 | printed to two decimals by the primary implementation |
| Paper-mill diff CI lower | 0.01 | printed to two decimals by the primary implementation |
| Paper-mill diff CI upper | 0.01 | printed to two decimals by the primary implementation |
| Risk ratio (paper mill) | 0.01 | printed to two decimals by the primary implementation |
| Two-proportion z | 0.1 | large-magnitude normal-approximation statistic |
| MH Odds Ratio | 0.05 | iterative estimator; accumulation order differs between implementations |
| MH OR CI lower | 0.05 | iterative estimator; accumulation order differs between implementations |
| MH OR CI upper | 0.05 | iterative estimator; accumulation order differs between implementations |
| MH Risk Ratio | 0.05 | iterative estimator; accumulation order differs between implementations |
| Crude RR (MH table) | 0.01 | printed to two decimals by the primary implementation |
| Strata contributing | 0.5 | implementation printing precision |
| Cochran Q | 5 | sum over strata; accumulation order differs between implementations |
| Bootstrap: median lag diff (point) | 0.5 | implementation printing precision |
| Bootstrap: median lag diff CI low | not enforced | different RNG streams cannot be expected to coincide; reported as a consistency check only |
| Bootstrap: median lag diff CI high | not enforced | different RNG streams cannot be expected to coincide; reported as a consistency check only |
| Bootstrap: MAD diff (point) | 0.5 | implementation printing precision |
| Bootstrap: MAD diff CI low | not enforced | different RNG streams cannot be expected to coincide; reported as a consistency check only |
| Bootstrap: MAD diff CI high | not enforced | different RNG streams cannot be expected to coincide; reported as a consistency check only |
| Bootstrap: PM diff (point) | 0.01 | printed to two decimals by the primary implementation |
| Bootstrap: PM diff CI low | not enforced | different RNG streams cannot be expected to coincide; reported as a consistency check only |
| Bootstrap: PM diff CI high | not enforced | different RNG streams cannot be expected to coincide; reported as a consistency check only |

Bootstrap interval endpoints carry no pass or fail determination, by design: the two
implementations draw different pseudorandom sequences from the same seed, so their
endpoints differ by construction. Point estimates, which do not depend on the
resampling, are compared under the tolerances above.


## Comparison Table

| Estimate | PowerShell | Python (scipy) | Absolute difference | Tolerance | Agree? |
|---|---|---|---|---|---|
| Median lag, machine (days) | 491 | 491.0 | 0.0000 | 0.5 | YES |
| Median lag, comparison (days) | 488 | 488.0 | 0.0000 | 0.5 | YES |
| Mann-Whitney U | 274613105 | 274613105.0 | 0.0000 | 0.5 | YES |
| Mann-Whitney z | 8.391 | 8.3908 | 0.0002 | 0.01 | YES |
| Cliff's delta | 0.0545 | 0.0545 | 0.0000 | 0.001 | YES |
| MAD machine (days) | 267.5 | 267.5152 | 0.0152 | 0.5 | YES |
| MAD comparison (days) | 768.5 | 768.5251 | 0.0251 | 0.5 | YES |
| MAD difference (days) | -501 | -501.0 | 0.0098 | 0.5 | YES |
| MAD diff CI lower (days) | -514.1 | -514.0602 | 0.0398 | 0.5 | YES |
| MAD diff CI upper (days) | -488 | -488.0 | 0.0405 | 0.5 | YES |
| MAD ratio | 0.348 | 0.3481 | 0.0001 | 0.001 | YES |
| Welch t (Brown-Forsythe) | -75.246 | -75.2463 | 0.0003 | 0.01 | YES |
| Welch df | 37255 | 37254.8 | 0.1565 | 1 | YES |
| Paper-mill % machine | 71.12 | 71.1167 | 0.0033 | 0.01 | YES |
| Paper-mill CI lower, machine | 70.18 | 70.1799 | 0.0001 | 0.01 | YES |
| Paper-mill CI upper, machine | 72.04 | 72.0358 | 0.0042 | 0.01 | YES |
| Paper-mill % comparison | 9.14 | 9.1378 | 0.0022 | 0.01 | YES |
| Paper-mill CI lower, comparison | 8.9 | 8.9037 | 0.0037 | 0.01 | YES |
| Paper-mill CI upper, comparison | 9.38 | 9.3774 | 0.0026 | 0.01 | YES |
| Paper-mill diff (pp) | 61.98 | 61.9789 | 0.0011 | 0.01 | YES |
| Paper-mill diff CI lower | 61.02 | 61.0211 | 0.0011 | 0.01 | YES |
| Paper-mill diff CI upper | 62.94 | 62.9368 | 0.0032 | 0.01 | YES |
| Risk ratio (paper mill) | 7.78 | 7.7827 | 0.0027 | 0.01 | YES |
| Two-proportion z | 144.12 | 144.1162 | 0.0038 | 0.1 | YES |
| MH Odds Ratio | 2.311 | 2.3112 | 0.0002 | 0.05 | YES |
| MH OR CI lower | 2.15 | 2.1505 | 0.0005 | 0.05 | YES |
| MH OR CI upper | 2.484 | 2.4840 | 0.0000 | 0.05 | YES |
| MH Risk Ratio | 1.431 | 1.4307 | 0.0003 | 0.05 | YES |
| Crude RR (MH table) | 7.783 | 7.7827 | 0.0003 | 0.01 | YES |
| Strata contributing | 23 | 23.0 | 0.0000 | 0.5 | YES |
| Cochran Q | 248.2 | 248.1870 | 0.0130 | 5 | YES |
| Bootstrap: median lag diff (point) | 3 | 3.0 | 0.0000 | 0.5 | YES |
| Bootstrap: median lag diff CI low | -17 | -16.0 | 1.0000 | n/a | ~approx (bootstrap) |
| Bootstrap: median lag diff CI high | 40 | 42.0 | 2.0250 | n/a | ~approx (bootstrap) |
| Bootstrap: MAD diff (point) | -501.01 | -501.0098 | 0.0002 | 0.5 | YES |
| Bootstrap: MAD diff CI low | -563.32 | -562.4786 | 0.8414 | n/a | ~approx (bootstrap) |
| Bootstrap: MAD diff CI high | -425.78 | -420.0861 | 5.6939 | n/a | ~approx (bootstrap) |
| Bootstrap: PM diff (point) | 61.98 | 61.9789 | 0.0011 | 0.01 | YES |
| Bootstrap: PM diff CI low | 57.85 | 57.8335 | 0.0165 | n/a | ~approx (bootstrap) |
| Bootstrap: PM diff CI high | 66.5 | 66.6752 | 0.1752 | n/a | ~approx (bootstrap) |

## Bootstrap Note

Bootstrap CI endpoints differ between implementations because Python's
`numpy.random.default_rng` and PowerShell's `[System.Random]` produce different
pseudorandom sequences even with the same seed. The point estimates (which do not
depend on the RNG) should agree exactly. CI endpoint differences of < 15 days for
lag measures and < 2 percentage points for proportions are expected and do not
indicate an error in either implementation.



VALIDATION VERDICT: PASS

## Scope of this validation

This resource compares an independent numpy/scipy implementation against the primary
PowerShell implementation for the statistics listed above: the crude timing and dispersion
comparison, the co-coding proportions, the Mantel-Haenszel point estimates, and a
2,000-replicate cluster bootstrap.

It does **not** constitute a second independent implementation of every analysis in the
manuscript. Three analyses introduced during revision are implemented once only, in Python,
and are therefore not cross-validated here:

- the exact-publisher matched dispersion analysis (5,119 pairs), `matched_dispersion.py`;
- the citation rematching on citations accumulated through the retraction year (4,747 pairs),
  `referee_reanalysis.py`;
- the notice-cluster bootstrap for the Mantel-Haenszel estimates, `mh_cluster_bootstrap.py`.

Section 3.10 of the manuscript is worded to reflect this limitation. For those three analyses
the assurances offered are different in kind: deterministic seeds, permutation and
multiple-seed stability checks reported in the manuscript, and full source in Online
Resource 3 for third-party re-execution.

## Which bootstrap intervals the manuscript reports

The bootstrap rows above are the cross-implementation comparison at 2,000 replicates. Table 6
of the manuscript instead reports 10,000-replicate intervals from `t50_bootstrap_10k.csv`,
computed after a clustering correction: records whose retraction DOI is blank or the literal
string "unavailable" are treated as singleton clusters rather than sharing one artificial
cluster, which raises the cluster count from 57,064 to 61,135.

| Estimate | Point | Table 6 interval (10,000 replicates) | MC SE of endpoints |
|---|---|---|---|
| Median lag difference (days) | 3 | -17 to 43 | 0.00, 0.33 |
| Mean-absolute-deviation difference (days) | -501.01 | -563.22 to -422.96 | 0.23, 0.44 |
| Paper-mill co-coding difference (pp) | 61.98 | 57.80 to 66.59 | 0.03, 0.04 |

Point estimates are identical under every replicate count and clustering variant, because they
do not depend on resampling. Where this file and Table 6 differ on an interval endpoint,
**Table 6 is the reported result.**

Validation regenerated 2026-07-31 directly from `validate_stats.py`; the tolerance table above
is emitted from the same list the comparison applies, so documentation and enforcement cannot
diverge.