A short progress note from the assessment of imaging neuroscience’s foundations, written from the methods side: what of it carries over to testing and research assessment in general. No conclusions about the field; those wait for completion.

What emerged as a tool

  • Inference reconstruction as a model. From a chosen paper we list every proposition and every step between them, and give each step a type: implies, requires, excludes, limits, asserts as the only option. From that, a diagram is generated together with the list of unsupported steps, unstated premises and exclusions never performed. Tried on four reconstructions.
  • Forks. Every point where the authors could have chosen otherwise — a different reference signal, threshold or control — is recorded as a fork with what the other branch would imply. That is where the unspoken assumption usually lives.
  • Statistics audit. Seven rules asking: is the test’s assumption checked? is the null stated? does the error control match what was actually tested? does the effect carry an uncertainty? was the threshold derived or chosen? can the number be reproduced? — and one question above them all: could the test have failed had the claim been false?
  • Two strata of sources. Careful methodological papers and ordinary applied practice are rated separately, so that a selection of careful papers does not turn into a verdict on the whole field.

Turned on ourselves

We ran the same rules on our own numerical projects. For one of our acceptance gates — the threshold for the empirical complexity check of an algorithm — the threshold turned out not to be derived, and the gate as written would reject a correct implementation over part of its range. The fix we propose is a change of measurement range, not a wider threshold. The record is in the proposal for a new criterion in our reference-implementation doctrine.

Next

A short piece about one entry condition most studies cannot verify awaits the author’s reading. A technical article on near-infrared imaging (physical roots and technology) is in progress. The full assessment documentation and a public repository with the whole trail follow.