Testing as invariant of engineering — a physicist’s perspective

Testing as an Invariant of SW Engineering: A Physicist’s Perspective

After years in the field of systems testing—dominantly software and enterprise systems—I have concluded that, with a background in physics, I view testing quite differently than most colleagues who encountered it through computer science studies. Physics is a discipline in which (if it functions as it should) hypothesis, interpretation, and experiment motivate and require one another. Measurement is at the core; the way an experiment is constructed simultaneously mirrors the level of understanding of the problem, as well as the method of data acquisition and evaluation—the conversion into what is popularly called a „physical quantity.“ The value of a physical quantity is meaningful only when linked to a measurement procedure and a unit. This same principle applies without modification to software quality. However, this does not mean it is appropriately represented in software projects. As a seasoned practitioner, I can say I have very rarely encountered a test architect or test analyst being properly present as early as the „inception“ phase of the entire „adventure.“ This regularly leads to such projects truly becoming adventures in the sense of an adrenaline sport.

Measurement as a Fundamental Act

Kelvin’s dictum is unequivocal: „When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it… your knowledge is of a meagre and unsatisfactory kind.“Quality that cannot be measured is an opinion. Quality measured according to defined criteria is an engineering result. Testing in engineering disciplines represents—though identical in principle—something slightly different than in scientific work (at least if we are talking about first-class work rather than mere repetition, but that is a topic for another time). A Test Architect „sees“ this best during the test design phase, where a holistic view and a broader risk assessment are required. We will certainly devote more observations to this in other posts; it is a fundamental topic that is not particularly well-handled even in common software literature.From this perspective, test analysis is the act of deriving measurable Test Targets from the Test Basis—requirement specifications, architectural contracts, interface definitions, etc. A Test Target is the engineering counterpart of what is called an „observable“ in modern physics (especially with reference to quantum mechanical usage): a precisely specified property of the system for which a metric can be coherently defined and a pass/fail criterion subsequently formulated. Without this derivation step, testing becomes a random exploration, and the results cannot be reproduced.

Coverage as a Sampling Problem

The theory of coverage in testing mirrors sampling theory in experimental science. The space of possible inputs for any non-trivial software system is astronomically large. The state space of a system—when using a bottom-up „from the code“ approach—grows exponentially. The involvement of third parties in the form of today’s popular machine learning systems, such as Agentic LLM-based „A.I.“, does not improve the situation regarding possible failure modes; on the contrary, it adds degrees of freedom that we do not have under control. Exhaustive testing in the sense of full coverage—both of inputs and of system states corresponding to empirically relevant functionalities, including all possible boundary, sub-boundary, and cross-boundary states—is combinatorial suicide. While automation makes repetitive regression tests cheaper and relatively eases testing when we have a well-written test script for a functionality (data parameterization), it requires maintenance and control over environment/dataset transitions. Again, generative tools radically lower the cost of producing scripts and sometimes mocking data, but not necessarily the correctness of coverage at the functional level (semantics is a rascal), and they produce extra work on the side of test suite and test data maintenance. The task of the test analyst (in cooperation with other project players) is therefore to select a finite, representative sample that provides statistically reliable coverage of the data input space and the system state space, which are closely interrelated.

The Invariant of Testing

Testing—as Measurement—is invariant across methodologies, much like the laws of physics do not depend on the methodology chosen for measurement (though the way they are formulated might, but that is for another post). A function either behaves according to the specification, or it does not. A sprint boundary does not change this. An architectural pattern does not change this. What changes between methodologies is the cadence at which measurement is performed and the scope of the test basis available at each measurement point. This is likely obvious to anyone experienced in large-scale SW projects.

The engineering-managerial consequence we would like to highlight: the cost of fixing defects rises steeply with the distance between the point of introduction and the point of detection. This is not a management principle—it is a consequence of the structure of software artifacts. The economic incentive for the early engagement of a Test Architect and subsequently a Test Analyst in the team is thus quite clear. One can even attempt to measure it in a financial sense, which in my experience rarely happens, as tolerance for bugs has increased over the last two decades, while tolerance for delivery speed and costs has worsened. Nevertheless, the incentive for early test analysis follows directly from the outlined structure.