A check that ran once is easy to pass off as proof. In a single day, in three different projects, we hit the same thing three times: the measurement was right, the conclusion was not, because one sample stood in for a whole distribution — or because the check had no way to fail. The three cases, from physics, numerical mathematics and bibliometrics, share one structure, so we describe them together.
1. One segment of data: a false exception to a correct rule
While reproducing the first gravitational-wave detection (GW150914) from public data, we checked a statement from the guide the LIGO and Virgo collaborations published with the data: a Fourier transform of un-windowed data is swamped by leakage from low frequencies. As a negative control we took one four-second segment from each detector, 300 seconds before the event, and compared the power around 500 Hz without and with a window.
Result: Livingston 1566× more power without the window, Hanford only 2×. So we proposed narrowing the wording: „holds for Livingston, not for Hanford in this segment“.
Before writing that down, we repeated the check on all 1023 non-overlapping segments of both files. The picture reversed:
| detector | 5th percentile | median | 95th percentile | segments above 10× |
|---|---|---|---|---|
| Hanford | 2.0× | 23× | 176× | 67 % |
| Livingston | 345× | 1032× | 2316× | 100 % |
The guide’s rule holds for both detectors. Our „Hanford 2ד was the segment at the fifth percentile. Had we drawn the conclusion from it, we would have published a false exception to a correct rule. Moreover: the same operation on the same instrument on the same day gives anything from 2× to 2300× depending on the detector and on which four seconds you pick — three orders of magnitude — and for a different reason in each detector (in Hanford the jump between the segment’s first and last sample; in Livingston not yet explained).
LIGO physicists have long known this: the noise spectrum is never estimated from one transform, but as a median over hundreds of segments.
2. A reference computation that could not disagree
In a public reference implementation of the Fourier transform (lege-artis/fourier) we measure accuracy against a computation in higher precision (long double). On a MacBook the probe reported zero error for every transform size.
The zero was not a result. On Apple Silicon processors long double is the same 53-bit type as double — the „more precise“ reference was the same computation as the code under test and could not disagree with it. A perfect score from an oracle that cannot differ is a silent instrument failure.
Fix: the probe now first checks that the reference really has more bits (LDBL_MANT_DIG), and if it does not, it refuses to measure and exits with an error. The reference is then computed by an independent library (mpmath, 40 significant digits), so the result does not depend on the platform. On a machine where long double has 113 bits, the error is 2 to 17 units of rounding depending on transform size — not zero; where both references can be compared, they agree exactly.
3. A statistical test that counted the same people twice
In an assessment of the foundations of brain imaging we measured whether papers describing a method’s limits are cited by the application literature too. The result looked strong: a statistically significant difference for three of seven papers, even after correction for multiple testing.
But the test counted authors who cited both compared papers in both cells of the table. For one method family that was roughly a third of the citers — and the significance was inflated by it. So we downloaded the full set of 23,390 citing works, stored it with checksums, and repeated the test with disjoint cells (every citer in exactly one). The ratios reproduced to the second decimal — the data were fine. The significance was not: after correction (Holm) a single paper survived (p = 0.002), two remained suggestive (p = 0.10), and we withdrew the original p-values.
The common structure
In all three cases the measurement was right. What was wrong was what we inferred from it, and each time for the same reason: one sample stood in for a distribution, or one thing stood in for two independent ones. One segment for 1023 segments. One computation for two (the tested and the reference). One citer for two (in both cells).
What do we take from this in practice? Three conclusions, each modest and well supported.
- A negative control run once is an anecdote. Only over the whole population (all segments, all configurations, all runs) does it become a distribution — and only then a control. Report the range, not one number.
- An oracle must be able to disagree. Before you trust a check, verify that it has a way to fail: that the reference is truly independent (a different computation, a different library, a different precision) and that it really fails on a known faulty case. A perfect result is a reason for suspicion, not relief.
- A statistical test holds only if its units are truly separate. Overlap between groups (the same user, the same test, the same data set in two categories) is not a small inaccuracy — it can manufacture significance that does not exist.
References
- Abbott, B. P. et al. (2020). A guide to LIGO–Virgo detector noise and extraction of transient gravitational-wave signals. Classical and Quantum Gravity, 37, 055002. https://doi.org/10.1088/1361-6382/ab685e
- Harris, F. J. (1978). On the use of windows for harmonic analysis with the discrete Fourier transform. Proceedings of the IEEE, 66(1), 51–83. https://doi.org/10.1109/PROC.1978.10837
- Weyuker, E. J. (1982). On testing non-testable programs. The Computer Journal, 25(4), 465–470. https://doi.org/10.1093/comjnl/25.4.465
- Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70. https://www.jstor.org/stable/4615733
- Gravitational Wave Open Science Center: GW150914, GWTC-1, version 3. https://gwosc.org/eventapi/html/GWTC-1-confident/GW150914/v3/ (data CC BY 4.0)
- OpenAlex — open bibliographic catalogue (data CC0). https://openalex.org
Our reference documentation (working versions in .md): memo „one segment is an anecdote“ and GW150914 results (fourier/examples/gw150914/); the false-zero finding (fourier/_audit/TWIDDLE-… §6); neuro-assessment Part 2 §3h (A7d → A8, withdrawn p-values), data with checksums. Currently in a private working repository; publication is in preparation.
Forthcoming. A text on why a deterministic test built on simplified but unambiguous domain semantics has particular value in the age of development with language models — and what code is actually the code of, for the developer, the analyst and the user. We will also publish the code and data for all three cases.
Method and limits. All three cases are our own errors caught before publication, not criticism of others‘ work; in the first case the published statement turned out to hold. Numbers come from computations on 2026-10-11 (GW150914: GWOSC v3 data; fourier: MacBook Apple Silicon and Linux x86-64 sandbox; neuroscience: OpenAlex snapshot 2026-10-11T11:52Z, two independent snapshots identical). Text and code generator: Claude language model (Cowork), 2026-10-11 to 12; oracles: LIGO–Virgo guide, mpmath, disjoint Fisher test; domains: GWOSC v3 data, IEEE 754, OpenAlex as of the stated date.