The defining feedback loop in software development project setup is not between developer and test suite. It is between the person writing a feature (analyst, developer, tester) and the person who will use it. Everything else — unit tests, integration tests, code review, CI pipelines — exists to approximate and translate within the loop, when direct contact is not possible or not fast enough or – in case of enterprise systems it is always a case – semantic distance between Need of future user and code/data/outputs is so big and context is so wast/specific, that nobody could do it on her/his own. One of trickiest moments of today s „AI revolution“, which I prefer to frame as rapid development LLM based ecosystem fast evolution, is, that it promises to make a leap even in this leap, where arguments are almost solely trivial or specific – in such a case semantics plays just weak constraint role.  I d like just draw your attention to problems hidden behind these assumptions. IMHO game stay the same, some rules are changing but in the core of the „translation-needs-to-the-SW“ business stays „There is no such thing as vibe understanding“  I would disagree with Curt in many particular topics, but he well said most important message I guess

The two-question invariant

When a test engineer asks „how should I test this feature,“ the question is structurally identical to „what is the core functionality (or non/functional requirement, terrible wording, but common used) and why does it matter to the customer eg. mission?“ These are not two questions — they are the same question expressed from two positions in the process. A test that cannot be derived from a clear answer to the second question is not a useful test. Beware of V model! A feature description that cannot be turned into a test is not a usable specification.

This is not a testing observation. It is a communication observation. The function of the test design review — in any methodology — is to expose misalignments in the shared mental model of the system before those misalignments are encoded into code or test data. In the language of the Testing we distinct clearly between validation and verification goals, even in practice, especially when talking with those who likes :KISS: approach (Keep it Simple as Stupid) we dunno catch that phrase. When a test engineer cannot formulate a test for a requirement, that failure of formulation is a signal: the requirement contains a hidden assumption, an ambiguity, or a contradiction. The test design process is a structured protocol for extracting these hidden properties from natural-language descriptions.

Errors are easy to correct. Conceptual problems are not.

An error is a localised deviation: a wrong calculation, a missing null check, an off-by-one. Static analysis, automated tests, and linters catch errors reliably. Their detection cost has been declining for decades and continues to decline. LLM-assisted code generation would be an accelerating factor even further — the syntactic quality of generated code is, generally speaking, high, and common error patterns are well-represented in training data.

A conceptual problem is a deviation in the model of the system: a feature that correctly implements a wrong interpretation of a requirement; a data structure that accurately represents a domain concept that turns out to be the wrong concept; an integration pattern that coherently connects two services according to a contract that was never actually agreed. Static analysis cannot detect these. Automated tests cannot detect them unless the tests were written against a correct specification — and if the specification was wrong, the tests are wrong too, and they will pass.

This asymmetry — errors are findable, conceptual problems are not — is the persistent structural challenge of software quality. It does not diminish with tooling improvement. It intensifies. Faster delivery cycles reduce the time available for the communication that would surface conceptual problems before they are encoded. AI-assisted coding tools increase the throughput of code generation without increasing the precision of requirement understanding. The ratio of conceptual-problem risk to error risk increases.

AI as amplifier — not as specification validator

LLM-based coding and documentation tools are, at their current capability level, high-quality amplifiers of what is already in the prompt context. They are excellent at generating syntactically correct, stylistically appropriate code that matches the intent expressed in the prompt. They are poor validators of whether the intent expressed in the prompt matches the intent held in the customer’s head.

This is not a criticism — it is a description of the tool’s operating range, representing what they are, by principle. Bit of understanding of the math behind Transformers and training process would correct false expectations so much projected to such a tools. But that s topic for another post. A signal amplifier amplifies signal and noise equally. If the input to an LLM coding tool contains a conceptually wrong specification, the output will be a larger, more refined, better-formatted version of the conceptually wrong specification. The error detection rate for implementation errors will be high. The error detection rate for conceptual errors will be low or zero. The test engineer’s role in this environment is not to compete with the tool on error detection. It is to maintain the discipline of asking „does this specification correctly reflect what the customer needs?“ — the question the tool cannot answer.

Short feedback loops as quality infrastructure

The practical implication for team communication: the feedback loop between „what are we building and why“ and „how will we verify it“ must be kept shorter than the loop between „writing code“ and „deploying it.“ In a sprint-based team using AI coding tools, this means the test analysis meeting (or equivalent exchange) happens at story definition, not after code completion. The developer, the test engineer, and the product owner are in the same room (physical or virtual) when the acceptance criterion is written, because that is the moment when the conceptual model can still be corrected at minimum cost.

The argument is not philosophical. It is economic. The cost of correcting a conceptual problem discovered during acceptance testing is an order of magnitude higher than the cost of correcting it during story definition. In a team generating high volumes of code via LLM tools, the volume of code produced per unit of communication investment increases — which means the cost-per-defect of conceptual problems increases proportionally. The communication investment does not scale with code volume. The testing investment must. That s warning already well known to the folks with dirty hands from all project deliverables, not necessarily fully understandable to the higher management. I m proud to stay on my one leg in the dirty, muddy waters of real delivery and stay in touch with visions on the other one. But without grounding of the decisions in real world, things become messy with any technology to be pushed into.

References

  • Deming, W.E. — Out of the Crisis (1982). MIT Press.
  • DeMarco, T., Lister, T. — Peopleware: Productive Projects and Teams, 3rd ed. (2013). Addison-Wesley.
  • Beck, K. — Extreme Programming Explained: Embrace Change, 2nd ed. (2004). Addison-Wesley.
  • Weinberg, G.M. — The Psychology of Computer Programming (1971 / 1998 Silver Anniversary ed.). Dorset House.
  • Wiegers, K., Beatty, J. — Software Requirements, 3rd ed. (2013). Microsoft Press.
  • Feathers, M. — Working Effectively with Legacy Code (2004). Prentice Hall.