In November 2020, in the middle of the pandemic, the physicist Catherine Beauchemin gave a public lecture at the Perimeter Institute from her living room in Toronto. Her field is one she named herself, somewhat to position it for grant committees, as she admitted: virophysics — writing down how a virus infection spreads from cell to cell as equations rather than as a paragraph of text. The lecture was about trust in health research. The part I keep returning to is a story she told against her own work.

The same virus, different from itself

Her group had built a model of influenza infection in cell culture: how long an infected cell stays silent before producing virus, how fast it produces, how long it lives, how quickly free virus loses its infectivity. The model did something models rarely get to do in biology — it predicted the outcome of a competition experiment between a normal strain and a drug-resistant mutant before the experiment was run, and the prediction held.

So they ran the method again, with a new mutant and the same normal strain. And the normal strain turned out to differ more from itself, between the two experiments, than it differed from the mutant they were trying to study. Several parameters were properties of the batch of cells, not of the virus. The variation between experiments was larger than the effect under investigation.

A reviewer rejected the paper. The objection, paraphrased, was that variation of this kind should not exist if proper technique is used — perhaps two different people had done the experiments?

Jane

Her answer was a thought experiment, and it is the clearest statement of the problem I have heard.

Jane runs the comparison of two drugs alone, on one plate, on one day, with one batch of cells. The scatter is small and drug A is clearly better than drug B. Another lab runs the same comparison with three people, three batches, different reagents. The scatter is large and the difference is not significant.

These two studies do not contradict each other. They answer different questions. Jane has shown that the drug works on Mary between nine and ten in the morning, when Jane is measuring. The second lab has asked something closer to what we actually want to know. If Greg repeats Jane’s experiment on Bob, he may well get the opposite — and it will look like a failed replication, when it is simply a third narrow question. Both results are correct. Their range of validity is far narrower than anyone reading them would assume.

What the reviewer called proper technique — same person, same day, same instrument — Beauchemin compared to accepting the mass of a particle only when Jane is the one measuring it. Redefine what counts as variability and you have redefined what counts as a result.

What it looks like in a hospital

Her clinical example was a collagen sponge soaked in the antibiotic gentamicin, implanted during colorectal surgery to prevent wound infection. By 2010 it had been approved in 54 countries and used in more than a million patients since 1985. Then a randomised trial across 39 hospitals in the United States, 602 patients, with outcomes judged by people who did not know who had received the sponge, found wound infections in 30.0% of patients with it and 20.9% without. The treatment designed to prevent infection was associated with more of it. The trial was paid for by the manufacturer.

In the lecture the practice rests on an earlier single-centre trial — Jane’s experiment, in clinical form: precise, favourable, and valid for the place where it was done. The multicentre trial was the second lab. Neither is dishonest. Only one of them answers the question the surgeon is actually asking.

So the answer is bigger studies?

That is the tempting conclusion, and it is wrong in an instructive way.

In 2021 a group of statisticians looked at estimates of COVID-19 vaccine uptake in the United States. A survey collecting about 250,000 responses a week overestimated uptake by 17 percentage points. Its margin of error was tiny — around the wrong number. A conventional panel of about 1,000 people a week, drawn with care, got it right. The authors showed that the huge survey had the accuracy of a simple random sample of about ten people. Their conclusion was that trying to make up for quality with quantity is a mathematically losing proposition.

The reason is simple once you see it. The error of an estimate has two parts. The random part shrinks as the sample grows. The part that comes from who ended up in the sample does not shrink at all. A bigger sample narrows the interval around whatever number the selection produced — and adds a confidence the number has not earned.

The pandemic produced the matched pair. A 36-patient study of hydroxychloroquine from Marseille, published in March 2020, shaped treatment around the world; it has since been retracted, over among other things how its control patients were chosen and the fact that the viral tests were not run the same way in the two cities involved. Two months later The Lancet published an analysis of 96,032 patients from 671 hospitals on six continents. It was retracted two weeks later, because nobody could verify that the data existed. One study too small, one enormous, the same fate. Size was never the variable that mattered.

Where a physicist comes in

Beauchemin’s thesis is that many of these problems start with hypotheses stated in words. “The mutation slows the virus down” cannot be embarrassed by data the way an equation can. Write the same idea as a model and it immediately asks which parameter, by how much, and how you would tell that from batch-to-batch noise — and it tells you when your understanding has a hole in it.

Her group has done exactly this to one of virology’s most ordinary measurements: the standard way of reporting how infectious a virus sample is relies on approximations that are biased and often miscalculated, and every comparison built on it carries that bias forward unnoticed. That is not a scandal. It is a calibration problem, and physics has a long habit of treating calibration as part of the result rather than a chore that precedes it.

In the lecture she goes further: if a solution is found, it will be found within physics. Out of context that sounds like a discipline speaking for itself. In the context she describes — a field that trains its practitioners with almost no mathematics, states its hypotheses in words, and treats variation between experiments as a fault of technique rather than as a quantity to be measured — I find it hard to disagree with any step of it. The tools for telling a real effect from batch-to-batch noise exist, and they are the everyday tools of physics.

Physics is not immune, and its better practitioners say so. Bedřich Velický, a Prague solid-state theorist who has worked with electronic-structure calculations for more than fifty years, writes of computational physics: “Some curve always comes out. The computer creates a suggestive and alluring fictional world, which ought to be isomorphic — with what, exactly?” Agreement with reality is therefore checked in two stages, and “both stages are mandatory” (Velický 2016, pp. 190–191, our translation). Beauchemin asks nothing of virology that physics does not ask of itself.

Where Jane was standing

In an earlier piece here, From Where Is It Seen, I wrote about the vantage point every measurement quietly carries. Jane is that vantage point with a name. A result is not wrong for being narrow. It becomes wrong when it is read as if it had been taken from nowhere.

The Prague philosopher Pavel Kouba puts it more generally, and it holds far outside the laboratory: “The more unambiguous the relations we arrive at in a given isolated context, the more problematic their use can be in grasping concrete reality” (Kouba 2016, p. 63, our translation). Jane arrived at a very unambiguous relation. Mary, on Tuesday between nine and ten, is her isolated context.

So the question worth asking of any study — of a drug, a therapy, a test suite — is not first “how many?” It is “who measured, on what, on which day, and how much would it have moved if someone else had?” If that last number is larger than the effect, the effect is not yet a fact. It is a very precise description of Tuesday.

Sources

  • Beauchemin, Catherine A. A. “A physicist’s adventures in virology.” Perimeter Institute Public Lecture, 4 November 2020. Video. (Paraphrased from the recording; not quoted verbatim.)
  • Paradis, E. G., et al., incl. C. A. A. Beauchemin. “Impact of the H275Y and I223V Mutations in the Neuraminidase of the 2009 Pandemic Influenza Virus In Vitro and Evaluating Experimental Reproducibility.” PLoS One 10(5), 2015, e0126115.
  • Cresta, D., et al., incl. C. A. A. Beauchemin. “Time to revisit the endpoint dilution assay and to replace the TCID50 as a measure of a virus sample’s infection concentration.” PLoS Computational Biology 17(10), 2021, e1009480.
  • Bennett-Guerrero, E., et al. “Gentamicin-collagen sponge for infection prophylaxis in colorectal surgery.” New England Journal of Medicine 363(11), 2010, 1038–1049.
  • Bradley, V. C., Kuriwaki, S., Isakov, M., Sejdinovic, D., Meng, X.-L., Flaxman, S. “Unrepresentative big surveys significantly overestimated US vaccine uptake.” Nature 600, 2021, 695–700.
  • Gautret, P., et al. “Hydroxychloroquine and azithromycin as a treatment of COVID-19.” International Journal of Antimicrobial Agents 56(1), 2020, 105949. Retracted.
  • Mehra, M. R., et al. “Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis.” The Lancet, 2020. Retracted 5 June 2020.
  • Kouba, P. “Pravda v rozumění” [Truth in understanding]. In K. Trlifajová, J. Frei et al., Spor o pravdu [The Dispute over Truth], Filosofia, Prague 2016, pp. 43–66. Czech only.
  • Velický, B. “Pravdivost a hledání pravdy v praxi řadového fyzika 21. století” [Truthfulness and the search for truth in the practice of a rank-and-file physicist of the 21st century]. Ibid., pp. 149–195.