Neuroscience assessment — technical notes · 3

The second note ended with a tidy story. The naive keyword search for „PET“ returned oncology; we added a field filter; the count fell from 124 436 to 21 942 and the top of the list became neuroimaging. Filter added, contamination gone. The story is true. The number is not evidence for it.

What was actually changed

Writing the metadata file for the second snapshot — the file that records every query URL verbatim — meant reading the URLs side by side. Between snapshot A0 and snapshot A1 the search strings were not the same. For PET, "PET imaging" had become "PET". For fMRI, "BOLD signal" had become "BOLD". For EEG, "event-related potential" had been added. Three of five queries changed at the same time as the filter. The working document had said „same terms, plus the filter“. It was wrong, and it was wrong in the direction that made the result look cleaner than it was.

So the drop from 124 436 to 21 942 is the joint effect of two interventions: a broader search string, which pushes the count up, and a field filter, which pushes it down. What each contributed is unknown. The one thing the pair of numbers cannot support is the sentence we had written under them.

Why this is worth a note

Because it is the smallest possible instance of the pattern the whole assessment is built to detect, and it happened to us on day one. A result, a stated cause, and an unstated second cause that the record did not surface until someone wrote the record down in full. Nobody intended to hide anything. The strings were improved between runs because the first run showed they were weak; the improvement was simply not carried into the sentence describing the second run. That is how most such things happen — not by fraud but by a description that lags the procedure.

In testing this has a name: a change set with two changes and one test. If the test flips, you do not know which change flipped it. The rule is one change per verification, or a factorial design when you cannot afford to serialise. The rule was known. It was not applied, because the second change did not feel like a change — it felt like a correction.

What caught it

Not a re-analysis. A clerical step. The assessment’s rule that every snapshot ships with a metadata file holding the exact URLs is there so that a reader can re-run the query. Its side effect is that the author has to look at the URLs. The mismatch was visible in the first ten characters after search:. A record kept for the reader’s benefit turned out to be the check the author needed.

A second error surfaced the same way. The filter was described as including „Experimental & Cognitive Psychology“. The data file’s own subfield column shows that the code we used, 3204, is „Developmental and Educational Psychology“; the one we meant, 3205, was never in the filter. The snapshot’s rows contradicted the snapshot’s description, and the rows are the record.

What happens now

Nothing is deleted or rewritten. The working document keeps the wrong sentence struck through, with a dated correction beside it, because the correction is part of the evidence about how the work was done. The next snapshot is designed to separate the two effects: the old strings under the new filter, the new strings without any filter, and the new strings under a filter that finally includes the subfield we meant. Four files, four digests. Only then does any sentence about „what the filter removed“ get written.

The assessment has a standing rule that it must not commit the errors it is looking for. This note is what that rule costs in practice: a public correction of a number nobody outside the project had yet relied on. It is cheap now. It is the same error, at a different scale, that costs a field its credibility when it is found in a paper cited three thousand times.

Next note: two indices, one question — how far OpenAlex and Semantic Scholar disagree, and what to do about it.