A pattern keeps showing up in teams that have adopted AI-assisted development. The reasoning sounds compelling at first: “The AI writes the code. Why do I still need to spend time on tests? Surely the AI can handle that too.”

This is a misunderstanding of what tests are for. Software failure modes are properties of the system — not of the author. Whether a function was written by a junior developer, a principal engineer, or a language model, the same class of bugs is possible: boundary conditions missed, edge cases unhandled, implicit assumptions unexamined. The mechanism of authorship is irrelevant. The failure modes are not.

But here’s the thing: the teams embracing AI testing aren’t wrong to try. The problem isn’t that they’re using AI for tests. The problem is how — and specifically, a failure mode that emerges when one model (or one context window) is responsible for all three artifacts at once: the requirements, the tests, and the implementation.

Tautological Testing

Call it what it is: tautological testing. A tautology is a statement that validates itself — true by definition, revealing nothing. A tautological test does the same thing. It’s a test that can’t fail, not because the code is correct, but because the test and the code were generated from the same misunderstanding.

Here’s how it happens in practice.

You have a requirement: “Apply a 10% discount for orders of $100 or more.” You feed that to an LLM and ask it to write both the implementation and the tests. The LLM produces this:

def calculate_price(amount: float) -> float:
    if amount > 100:          # misread: > instead of >=
        return amount * 0.9
    return amount

And the tests:

def test_discount_for_large_order():
    assert calculate_price(150) == 135.0   # passes

def test_no_discount_for_small_order():
    assert calculate_price(80) == 80.0     # passes

The test suite goes green. The CI pipeline is happy. But the model silently misread “or more” as “over” — and then wrote a test that confirmed its own misread. calculate_price(100) returns 100.0 when it should return 90.0. Nobody will find this until a customer with a $100 order complains that the discount never applies.

The model didn’t write a bug and then catch it. The model wrote a bug and then validated it.

Why This Is Different From the Human Case

When a human developer writes a buggy implementation and then writes tests, something usually breaks the loop. Another team member reviews the PR. A QA engineer tries the boundary case. A product manager runs a manual smoke test. The test author and the code author might be the same person, but they exist in a social and temporal context that creates friction — a gap where mistakes can be caught.

AI doesn’t have that gap. It has context. When you ask a model to implement a feature and test it in the same session, it reasons from the same internal representation of your requirements. The misunderstanding is consistent. The implementation reflects it. The tests reflect it. Everything agrees — and agreement is not the same as correctness.

This is the structural weakness of the single-context workflow. Not carelessness. Not low capability. A systematic, invisible alignment between the implementation and its verification that removes the very thing testing was supposed to provide: an independent check.

Breaking the Loop: A Two-Role Workflow

The solution isn’t to stop using AI for testing. It’s to stop giving the same model the same context for all three steps. Independence is the mechanism that makes testing meaningful. So the workflow has to enforce it.

Here’s a practical approach built on two explicitly separated roles:

Role 1: The Spec Verifier

Before any code is written, take your requirements to an LLM with a very specific mandate: find what’s wrong with this spec.

The prompt isn’t “help me implement this.” It’s “you are a domain expert and product engineer. Read this specification. Find edge cases I haven’t considered, ambiguities that could lead to different implementations, missing invariants, and boundary conditions that aren’t specified.”

Applied to our discount example, a good Spec Verifier might surface:

  • “Is the $100 threshold inclusive or exclusive?”
  • “What happens with a $0 order?”
  • “Does the discount apply to the pre-tax or post-tax amount?”
  • “Can discounts stack with other promotions?”
  • “Is there a maximum discount cap?”

None of these are implementation questions. They’re specification questions. And they’re exactly what a human reviewer would ask in a requirements meeting — but rarely does, because we’ve already mentally committed to a rough implementation by the time we’re discussing details.

The output of this step is a verified behavioral specification — a tightened, explicit description of what the system should do, with edge cases resolved and invariants stated.

For our example, the output might read: “The 10% discount applies to orders where amount >= 100. The threshold is inclusive. The discount applies to the gross amount before tax. Discounts do not stack.”

Role 2: The Independent Implementer

Now — and only now — take that verified specification to a fresh session. Different context, different prompt framing, ideally a different model. The mandate here is equally specific: you are a senior developer. Implement this specification using TDD. Write the test first. Watch it fail. Then write the minimal code to make it pass.

The critical constraint is that this session has no knowledge of any previous implementation ideas. It sees only the verified spec. Its test is derived from the specification directly — not from watching what the implementation does.

This produces tests that are genuinely independent of the implementation. If the implementation makes a wrong assumption, the test will catch it, because the test was written from a different starting point.

# Test written against the verified spec — before implementation
def test_discount_applies_at_exactly_100():
    assert calculate_price(100) == 90.0   # boundary is inclusive

def test_discount_applies_above_100():
    assert calculate_price(150) == 135.0

def test_no_discount_below_100():
    assert calculate_price(99.99) == 99.99

def test_zero_order():
    assert calculate_price(0) == 0.0

The first test — calculate_price(100) == 90.0 — would have caught the original bug immediately. The > versus >= mistake fails right at the boundary, exactly where the specification said the behavior should change.

What This Changes (and What It Doesn’t)

The two-role workflow doesn’t eliminate AI from the testing process. If anything, it uses AI more — for specification analysis, for test authorship, for implementation. The difference is in the architecture of how the AI is used.

What changes:

  • The model that generates tests doesn’t know how the implementation will be written.
  • The model that verifies the spec doesn’t know what the implementation looks like either.
  • Each role receives a bounded input and produces a bounded output.
  • The outputs are in tension with each other by design.

What doesn’t change:

  • The developer still owns the verified specification. They have to read the Spec Verifier’s output and decide which concerns are real, which boundary conditions matter, which invariants to enforce.
  • The developer still owns the decision to ship. Passing tests are evidence, not proof.
  • The refactoring step — the “make it clean” phase after “make it pass” — still requires judgment about the overall design that no single-step prompt will reliably produce.

The Deeper Principle

The two-role workflow is an instance of a broader principle: in systems that verify themselves, the verification is worthless.

This is well understood in security — you don’t audit your own code for security vulnerabilities, because the mental model that produced the vulnerability is the same one reviewing it. It’s well understood in finance — the accountant who makes the ledger doesn’t get to sign off on the ledger. It’s the foundation of peer review in science.

We just haven’t consistently applied it to AI-assisted development yet. The instinct is to reach for the same tool that’s already open, in the same chat session that already has context, and ask it to handle the next step. It’s fast. It feels seamless. And it produces the specific kind of error that’s hardest to detect: the error that looks correct because everything around it agrees.

AI-assisted engineering doesn’t mean “let the AI handle it.” It means shifting our focus from writing lines of code to verifying intent. The speed gain from AI is real. But speed without independent verification is not velocity — it’s accumulation of invisible debt.

Tests don’t care who typed the code. But they only mean something if someone other than the author decided what they should say.

This post is part of the AI Engineering Practice series — exploring what changes and what stays the same when AI enters the engineering workflow. Related: AI + TDD: A Shortcut to the Goal or a Loss of Insight?