Your AI Wrote 300 Tests. How Many Actually Assert Anything?

Coverage Is A Measure Of Execution, Not Verification

This distinction sounds pedantic until it costs you an outage.

A line counts as covered when the test run touched it. That is the entire definition. It does not require that the test looked at the result, compared it to anything, or would behave differently if the function returned garbage. You can reach 100% line coverage on a module with a test file containing zero assertions - just call every function and ignore what comes back.

Empirical work on AI-generated tests bears this out in a specific way: AI-produced test methods tend to be longer, with a higher density of assertions and lower cyclomatic complexity than human-written ones, and contribute coverage comparable to human tests. That sounds reassuring until you notice what higher assertion density does not guarantee - that the assertions are checking the thing that could break.

The guidance that has settled out of 2026 practice puts it plainly: quality of assertions matters more than quantity of lines covered, and a 60% coverage suite with thoughtful assertions catches more bugs than a 90% suite full of trivial checks.

The Four Failure Shapes

Across AI-generated suites the same four patterns recur. VibeDoctor tracks them as TST-002 through TST-005, which is a convenient way to name them.

All four produce green suites. Three of them raise your coverage number. None of them will catch the null that arrives from a third-party API at 2am.

What This Looks Like In Practice

Ask an assistant to "add tests for the checkout service" and you will often get something like this back. It is not a strawman - this shape is extremely common in Cursor and Copilot output.

// ❌ BAD - 100% coverage of applyDiscount, verifies nothing
import { describe, it, expect, vi } from 'vitest';
import { applyDiscount } from './checkout';

describe('applyDiscount', () => {
  it('applies a discount', () => {
    const result = applyDiscount(100, 0.2);
    expect(result).toBeDefined();          // true for 80, and for null, and for NaN
  });

  it('handles zero', () => {
    expect(() => applyDiscount(0, 0.2)).not.toThrow();
  });

  it('handles invalid input', () => {});   // empty body, always green

  it('calls the pricing service', () => {
    const spy = vi.fn().mockReturnValue(80);
    expect(spy(100, 0.2)).toBe(80);        // asserts the mock, not the code
  });
});

Four tests. Green. Coverage report says the function is fully exercised. Now ask what would happen if applyDiscount returned NaN for a negative rate, or silently produced a negative total for a discount above 1.0. Every one of these tests still passes.

The fourth test is the most instructive. It creates a mock, tells the mock to return 80, then asserts that it returned 80. It never imports the behaviour it claims to test. This is not a rare mistake - it is what a generator produces when it has been asked for a test and has no way to know what the function is supposed to guarantee.

How To Fix It

Rewrite so each test names a rule the code must obey and would fail if that rule broke.

// ✅ GOOD - each test states a rule and would fail if it broke
import { describe, it, expect } from 'vitest';
import { applyDiscount } from './checkout';

describe('applyDiscount', () => {
  it('subtracts the discount from the total', () => {
    expect(applyDiscount(100, 0.2)).toBe(80);
  });

  it('never returns a negative total', () => {
    expect(applyDiscount(100, 1.5)).toBe(0);
  });

  it('rejects a negative rate rather than returning NaN', () => {
    expect(() => applyDiscount(100, -0.1)).toThrow('Invalid discount rate');
  });

  it('keeps currency precision to two decimals', () => {
    expect(applyDiscount(10.99, 0.333)).toBe(7.33);
  });
});

The rewritten suite has fewer tests and lower coverage of incidental branches. It is dramatically more useful, because every line encodes a decision someone made about how money should behave.

The practical difficulty at scale is finding the first version across a few hundred generated test files. Tools like VibeDoctor's Vibe Check (vibedoctor.io) automatically scan your codebase for empty test bodies, tests with no assertions and happy-path-only suites, and flag specific file paths and line numbers. Free to sign up.

A Five-Minute Audit You Can Run Now

Before adopting any tooling, these three commands tell you most of what you need to know about a generated suite.

// Tests whose body is empty
// grep -rn "it(.*=> *{ *} *)" src/ tests/

// Test files containing no expect() at all
// grep -rLn "expect(" $(find . -name "*.test.ts" -not -path "*/node_modules/*")

// Ratio of assertions to tests - under ~1.5 is a warning sign
// echo "$(grep -rc 'expect(' tests/ | awk -F: '{s+=$2} END {print s}') assertions"
// echo "$(grep -rc '\bit(' tests/ | awk -F: '{s+=$2} END {print s}') tests"

If the second command returns any files, you have test files that cannot fail. Delete them or write them properly - keeping them is worse than having no tests, because they make the dashboard green.

Why This Matters More For AI-Generated Code

There is a compounding effect worth naming. A systematic review of 101 sources on AI-assisted coding quality found that QA is the most frequently overlooked dimension of AI coding workflows. The same tool wrote the implementation and the tests, from the same understanding of the problem - so a misunderstanding in the implementation is faithfully reproduced in the test that is supposed to catch it.

That is the trap in "100% coverage with AI-generated tests": the suite validates the implementation rather than the specification, which produces confidence without verification. Recommended practice for AI-generated code has drifted upward to 85%+ coverage against the 70-80% typical for human-written code - but only because the density has to compensate for the tests being written by the same process that wrote the bug.

Teams do report 40-70% faster test writing with AI generation, and that is a genuine gain. It is a gain in speed of production, not in assurance. Those need to be tracked separately.

FAQ

Sources

• Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and Coverage - structural characteristics of AI-generated tests, assertion density and coverage comparison.
• How to Test AI-Generated Code: QA Checklist for 2026 - assertion quality over coverage, and the 85% threshold for AI-generated code.
• Does AI-Generated Code Reduce the Need for Testing, or Demand More? - the 101-source systematic review finding QA is the most overlooked dimension.