In October 1969, at a NATO conference in Rome, a man from IBM named Joel Aron said something that has been quoted ever since, usually with a sentence missing.
„We are discussing two different things. One is debugging: the term we use for an individual finding errors in his own work within the design he created. The other is testing: the term applies to identifying and resolving problems that arise in the system as a whole rather than in individual components. Testing is done by groups, as an individual programmer lacks the total knowledge needed to test large sections of the system. The emphasis on testing should never be on individual modules and as long as system testing is confused with debugging we can expect test plans to be incomplete.“
The bolded clause is the one that usually gets dropped, and dropping it turns an argument about where to aim into a remark about org charts. Aron is not complaining that programmers are lazy. He is saying something structural: a person debugging his own design is looking for what he already suspects, and testing has to find what nobody suspects. The two activities point in different directions, and when you confuse them, the plan is incomplete — not through carelessness, but because the wrong person decided where to look.
Fifty-seven years later, that is still the unsolved part.
This is a long piece. It is the first of a series, and its job is to lay out where this discipline actually came from — who did the work, what the papers really say, and which of the arguments are still live. I have tried to get the dates right, because a field that demands traceable evidence for every test result has a bad habit of recounting its own history without any.
Before there was a discipline
Alan Turing gave a paper on 24 June 1949 at the conference marking the commissioning of EDSAC in Cambridge. „Checking a Large Routine“ is three pages, and its argument is this:
„the programmer should make a number of definite assertions which can be checked individually, and from which the correctness of the whole programme easily follows.“
It is routinely called the first paper on program verification. That is defensible as a claim about priority of the idea and misleading about everything else. Turing works one example — a flowchart computing n! by repeated addition — and does not present a general method. Two things are worth knowing if you ever cite it: the 1949 published text contains transcription errors, and the corrected version with commentary is Morris and Jones, „An Early Program Proof by Alan Turing,“ IEEE Annals of the History of Computing 6(2), April 1984, pp. 129–143. And the person who turned the single example into a general technique was Robert Floyd, in „Assigning Meanings to Programs“ (Mathematical Aspects of Computer Science, AMS, 1967, pp. 19–32), eighteen years later. Floyd’s inductive assertion method is the direct ancestor of Hoare’s {P} S {Q} triples, from „An Axiomatic Basis for Computer Programming,“ CACM 12(10), October 1969, pp. 576–583.
So the theoretical line starts with a case study, becomes a method eighteen years later, and becomes a logic two years after that.
The two conferences that named the problem
The NATO Science Committee convened two conferences: Garmisch, 7–11 October 1968, and Rome, 27–31 October 1969. Around fifty to sixty invited people each. The Garmisch report is generally credited with the first prominent use of „software engineering“ — chosen, by several participants‘ later accounts, as a deliberately provocative label implying that programming ought to be an engineering discipline and mostly was not.
The occasion was what the organisers openly called a software crisis: software late, over budget, unreliable, unmaintainable — and the problem getting more visible as hardware improved, because cheaper machines kept inviting more ambitious software.
What the reports say about testing is less programmatic than later summaries suggest, because most of it is transcribed conversation rather than agreed doctrine. Section 3.1.1 of the Rome report contains this exchange, which captures the state of the art in October 1969 better than any paraphrase:
Hoare: „One can construct convincing proofs quite readily of the ultimate futility of exhaustive testing of a program and even of testing by sampling. So how can one proceed? The role of testing, in theory, is to establish the base propositions of an inductive proof.“
>
Perlis: „Much of program complexity is spurious and a number of test cases properly studied will exhaust the testing problem. The problem is to isolate the right test cases, not to prove the algorithm.“
>
Dijkstra: „Testing shows the presence, not the absence of bugs.“
Perlis, in two sentences, states the problem this series is really about: the problem is to isolate the right test cases. Aron’s remark, quoted at the top, is in section 3.1.2 of the same report.
A note on the most-quoted sentence in the field
Dijkstra’s line exists in two textually distinct forms and they get conflated constantly.
The short version — „Testing shows the presence, not the absence of bugs“ — is a spoken remark in the Rome discussion transcript, published April 1970.
The famous version — „Program testing can be used to show the presence of bugs, but never to show their absence!“ — is a corollary in EWD249, „Notes on Structured Programming“, a privately circulated note dated August 1969, later republished as chapter 1 of Dahl, Dijkstra and Hoare, Structured Programming (Academic Press, 1972).
So if you have ever cited „Dijkstra, NATO 1969“ and quoted the version with the exclamation mark, you cited something that is not in those proceedings. It is a small thing. It is also the first of several places in this history where the story everyone tells and the document everyone cites are not the same object.
Royce, and the model he warned against
Winston W. Royce, „Managing the Development of Large Software Systems,“ Proceedings, IEEE WESCON, August 1970, pp. 1–9. Royce was then director of the Lockheed Software Technology Center in Austin.
He draws the sequential diagram — requirements, analysis, design, coding, testing, operations — and immediately writes: „I believe in this concept, but the implementation described above is risky and invites failure.“
The rest of the paper is five modifications to stop it failing. The most important is to do the whole thing twice: „arrange matters so that the version finally delivered to the customer for operational deployment is actually the second version insofar as critical design/operations areas are concerned.“ He also states, explicitly, that the design „must be tested by an entirely different group of test specialists working with a completely separate test plan“ — an early articulation of the independent test team, and a direct echo of Aron’s point from the previous year.
The word „waterfall“ does not appear in the paper.
Theory, 1975 to 1982
Three papers do most of the load-bearing work, and the field was substantially finished with its theoretical foundations by the early eighties.
Goodenough and Gerhart, „Toward a Theory of Test Data Selection,“ IEEE TSE SE-1(2), June 1975, pp. 156–173, put testing on a formal footing. They define what makes a test-selection criterion reliable (it consistently either reveals or fails to reveal an error, whichever data satisfying it you pick) and valid (if an error exists, some data satisfying the criterion will reveal it), and prove that a criterion with both properties is equivalent to a full correctness proof.
A correction, because I got this wrong in my own working notes and the point of this series is not to do that silently: Goodenough and Gerhart did not prove that common structural criteria fail these tests. They supplied the definitions that made the negative result statable. The impossibility results came from Howden, „Reliability of the Path Analysis Testing Strategy,“ IEEE TSE SE-2(3), May 1976, pp. 208–215 — no computable, practically applicable criterion satisfies both — and from Weyuker and Ostrand, „Theories of Program Testing and the Application of Revealing Subdomains,“ IEEE TSE 6(3), 1980, pp. 236–246 — the two properties are essentially mutually exclusive for a given criterion on given software. The chain is definitions, then non-computability, then mutual exclusion. It is worth citing correctly.
DeMillo, Lipton and Sayward, „Hints on Test Data Selection: Help for the Practicing Programmer,“ Computer 11(4), April 1978, pp. 34–41, named mutation testing and gave it two hypotheses: the competent programmer hypothesis (real bugs are small syntactic deviations from a correct program, not gross structural errors) and the coupling effect (test data that distinguishes a program from all its simple single-fault mutants will also distinguish it from programs with more complex errors). The second is asserted empirically, not proved. A year earlier, Hamlet, „Testing Programs with the Aid of a Compiler,“ IEEE TSE SE-3(4), July 1977, pp. 279–290, had arrived at a related idea by a different route.
And then the paper that matters most.
The oracle problem
Elaine J. Weyuker, „On Testing Non-Testable Programs,“ The Computer Journal 25(4), November 1982, pp. 465–470.
Her target is the oracle assumption — the unexamined premise, built into all prior testing theory including Goodenough and Gerhart, that the tester has some way of deciding whether a given output is correct. She shows it fails for a class of programs that is neither small nor exotic:
„Programs which were written in order to determine the answer in the first place. There would be no need to write such programs, if the correct answer were known.“
Scientific simulations. Numerical solvers. Compilers. Anything you build precisely because you do not already know the answer.
This is not an engineering gap that better tooling will close. It is an epistemic limit. For a large and important class of software, „the output matched the check“ and „the output was correct“ are not the same claim and cannot be made the same claim. Everything since — specified oracles, derived oracles, implicit oracles, metamorphic testing — is an accommodation to a limit Weyuker identified in 1982. The current survey of those accommodations is Barr, Harman, McMinn, Shahbaz and Yoo, „The Oracle Problem in Software Testing: A Survey,“ IEEE TSE 41(5), May 2015, pp. 507–525.
The practitioners
Running alongside the theory, a different literature was working out how testing is actually done by people.
Gerald Weinberg, The Psychology of Computer Programming (Van Nostrand Reinhold, 1971), is not a testing book, but egoless programming comes from it, and every subsequent review and inspection practice depends on defect-finding not being experienced as personal attack.
Glenford Myers, The Art of Software Testing (Wiley-Interscience, 1979), gives the field its most quoted definition:
„Testing is the process of executing a program with the intent of finding errors.“
Myers is explicit that this inverts the conventional definition of the time — „the process of demonstrating that errors are not present“ — which he calls psychologically self-defeating, because someone motivated to show a program works will unconsciously pick tests unlikely to break it. The book opens with the triangle problem: read three integers as the sides of a triangle, report whether it is scalene, isosceles or equilateral. Try it before reading on, is the instruction, and the point is to let you discover personally how many degenerate cases your honest first attempt misses.
Gelperin and Hetzel, „The Growth of Software Testing,“ CACM 31(6), June 1988, pp. 687–695, give the field its standard periodisation, and the dates are usually quoted loosely:
| Era | Period | Orientation |
|---|---|---|
| I | until 1956 | debugging — testing and debugging not distinguished |
| II | 1957–1978 | demonstration — show the software meets requirements |
| III | 1979–1982 | destruction — find errors |
| IV | 1983–1987 | evaluation — continuous quality measurement |
| V | 1988 onward | prevention — designing tests to prevent faults |
Note that era III is exactly where Myers’s definition sits, and that the paper was published in the year era V begins by its own account — which makes it partly a manifesto rather than purely a description.
Boris Beizer, Software Testing Techniques, 2nd ed. (Van Nostrand Reinhold, 1990), contributes the pesticide paradox — „every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual“ — and a five-stage model of how a tester’s own idea of the job matures, ending at a phase where testing is „not a discrete activity but a mental discipline that results in low-risk software without excessive dedicated testing effort.“
Read those three together and you can watch them arguing. Beizer’s phase 1 is Gelperin and Hetzel’s demonstration era. His phase 2 is Myers’s definition. His phase 3 and 4 are the field trying to grow past both.
Techniques that changed what was possible
Fuzzing. Miller, Fredriksen and So, „An Empirical Study of the Reliability of UNIX Utilities,“ CACM 33(12), December 1990, pp. 32–44. The origin is a thunderstorm: Miller was logged into a workstation over a dial-up line, noise on the line was injecting random characters into his session, and the programs were crashing rather than producing harmlessly garbled output. That struck him as worth investigating rather than shrugging at. They tested around ninety standard utilities across seven Unix variants; roughly a quarter could be crashed or hung by unstructured random input.
What makes fuzzing methodologically interesting is that it needs no oracle beyond „did it crash.“ It sidesteps Weyuker’s problem entirely by asking a question that has an unambiguous answer.
Model checking. Clarke and Emerson in the United States, Queille and Sifakis in France, independently and near-simultaneously in the early 1980s: exhaustive algorithmic verification of finite-state systems against temporal-logic specifications. Clarke, Emerson and Sifakis shared the 2007 ACM Turing Award for it.
Statistical usage testing. Mills, Dyer and Linger, „Cleanroom Software Engineering,“ IEEE Software 4(5), September 1987, pp. 19–25. Instead of testing to a structural coverage target, draw test cases probabilistically from a model of expected use, so that what you get out is a statistically certifiable estimate of field failure rate rather than a pass count.
Property-based testing. Claessen and Hughes, „QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs,“ ICFP 2000, pp. 268–279. State a property that should hold for all inputs of a type, let the tool generate inputs hunting for counterexamples, and shrink any counterexample to a minimal failing case. It is a direct descendant of a surprising result: Duran and Ntafos, „An Evaluation of Random Testing,“ IEEE TSE SE-10(4), July 1984, pp. 438–444, found that purely random testing could be more cost-effective per bug found than the structured partition schemes the theoretical literature had spent a decade elevating.
The metrics that do not measure what we think
Two results, published a year apart, should be read as a pair. They are among the most useful things the field has learned about itself.
Inozemtseva and Holmes, „Coverage Is Not Strongly Correlated with Test Suite Effectiveness,“ ICSE 2014, pp. 435–445. Five large Java systems, mutation-generated faults as ground truth. Once you statistically control for test suite size, the correlation between coverage and fault-detection effectiveness is weak to moderate — and stronger, more expensive criteria like branch coverage add little over plain statement coverage.
Their own conclusion is not that coverage is worthless. It is that coverage should not be used as a target or as a proxy for quality, because a large suite of redundant low-diversity tests scores well and catches little.
Zhang and Mesbah, „Assertions Are Strongly Correlated with Test Suite Effectiveness,“ ESEC/FSE 2015, pp. 214–224, supply the other half: the number of assertions predicts fault-detection effectiveness much better than coverage does.
Put them together and you have the cleanest statement of the problem I know:
Coverage measures reach. Assertions measure verification.
A line can be executed — and so counted as covered — without anything about the result being checked. The percentage in the report is a measurement of what ran, presented as if it were a measurement of what was confirmed.
That gap is this whole series‘ subject. It is also, in a different vocabulary, the gap this company exists to work on.
What the accidents taught
Therac-25. Leveson and Turner, „An Investigation of the Therac-25 Accidents,“ IEEE Computer 26(7), July 1993, pp. 18–41. Six known accidents, June 1985 to January 1987, radiation overdoses causing deaths and severe injuries.
The lessons go far past „there was a bug.“ The Therac-25 removed the independent hardware interlocks its predecessor had, on the theory that software could enforce the constraints alone. The fatal race condition required a rare sequence of rapid keyboard corrections by an experienced operator — which meant ordinary acceptance testing with inexperienced users was structurally unlikely to trigger it, and the manufacturer’s confidence that field use had proven the machine was a contributing cause rather than a mitigation. And the incident-investigation practices were themselves inadequate, which delayed both the fix and the withdrawal.
The methodological conclusion, and the foundation of software safety engineering as a field distinct from testing: accidents in complex software-controlled systems are rarely explained by a single root-cause defect. They are explained by an accumulation of organisational, process and design decisions, of which the defective line is the proximate trigger.
Ariane 5, Flight 501, 4 June 1996. Inquiry board chaired by Jacques-Louis Lions, reporting 19 July 1996. Break-up about 37 seconds after main engine ignition. An unhandled operand conversion — 64-bit float to 16-bit signed integer — in the inertial reference system’s alignment software. The critical detail is that the software was carried over essentially unchanged from Ariane 4, where the achievable horizontal velocity in that flight phase could never exceed the range. The assumption was true of one vehicle and false of the next.
And because primary and backup ran identical software, both failed within about 72 milliseconds of each other. Redundancy engineered against random hardware failure gives no protection at all against a deterministic software fault replicated across both channels.
Reused code carries no guarantee outside the envelope in which it was proven. That sentence is worth more than most test strategies.
Where the discipline actually stands
Read end to end, the record shows a field that has spent nearly eighty years rediscovering the same handful of hard limits under new names.
Weyuker’s oracle problem is Barr’s oracle problem is the reason metamorphic testing exists. Duran and Ntafos’s surprising result about random testing is why QuickCheck works. Aron’s 1969 insistence that testing and debugging are different activities is why independent test teams exist at all — and his warning about incomplete plans is why we are writing this series.
The theoretical core was substantially in place by 1982. Almost everything since has been either engineering around those limits — mutation testing at industrial scale, fuzzing, property-based testing, model checking — or documenting, empirically, how badly the field’s own proxy metrics track the thing they are assumed to measure.
And the folklore has been simplifying a real, more conditional history into cleaner and less accurate stories the whole time. That is the subject of the next piece.
The one after that takes up Aron’s actual question, which is still open: how do you decide what to test? The short answer, which surprised me when the evidence came in, is that this field’s strongest results are in its narrowest sub-problem — which tests to re-run given known history — while the broadest and most consequential question remains largely unvalidated heuristic and professional judgement. That asymmetry is not an embarrassment. It is a map of where the work is.
This is the first of a series on the Testing Overdose track. It was written from a research dossier in which every claim carries an evidence grade and seven items are listed as unverified and excluded from the text. Where I had previously got something wrong — the attribution of the structural-criteria impossibility results — the correction is left visible above rather than quietly repaired, for reasons the second piece will make obvious.