Mock Everything, Test Nothing: AI Testing Anti-Patterns

Why AI Tools Write Tests That Don't Catch Bugs

When you ask Cursor, Bolt, or Lovable to "add tests for this component," the AI optimises for tests that pass immediately without additional setup. The path of least resistance is to mock every external dependency (so no real database, API, or filesystem is needed) and to write a single test for the happy path (because that's the one described in the feature requirements).

The result is a test suite that achieves respectable line coverage - 70% or higher is achievable with this approach - but fails to catch the bugs that actually appear in production. According to a 2024 analysis by Testim.io, over 60% of production bugs occur in error-handling paths that were never covered by the test suite. These are precisely the paths AI tools skip.

The Google Testing Blog has documented this pattern extensively. Their research found that teams relying primarily on happy-path tests have a change failure rate 2-3x higher than teams with comprehensive error-path coverage, because regressions in error handling go undetected through the test suite and only surface when real users encounter real failures.

The Over-Mocking Anti-Pattern (TST-004)

Mocking is a legitimate tool - it's appropriate when you're testing a specific unit in isolation and want to control the behaviour of its dependencies. The problem is when mocking becomes reflexive: mock everything by default, without considering whether the interaction with the real dependency is exactly what should be tested.

// ❌ BAD - Over-mocking: every dependency is replaced, so nothing real is tested
import { vi, describe, it, expect } from 'vitest';
import { createOrder } from './orderService';

vi.mock('./database', () => ({
  db: { insert: vi.fn().mockResolvedValue({ id: '123' }) }
}));
vi.mock('./emailService', () => ({
  sendConfirmation: vi.fn().mockResolvedValue(true)
}));
vi.mock('./inventoryService', () => ({
  checkStock: vi.fn().mockResolvedValue(true),
  decrementStock: vi.fn().mockResolvedValue(true)
}));
vi.mock('./paymentService', () => ({
  chargeCard: vi.fn().mockResolvedValue({ success: true, transactionId: 'tx_001' })
}));

describe('createOrder', () => {
  it('creates an order', async () => {
    const result = await createOrder({ items: [...], userId: '1', card: {...} });
    expect(result.id).toBe('123'); // Tests only that the mock returned what we set up
  });
});

This test cannot detect: database constraint violations, inventory race conditions, payment service error responses, email delivery failures, or any bug in the interactions between these services. The test is testing the mocks, not the code.

// ✅ GOOD - Strategic mocking: only mock the payment gateway (external, paid service)
// Use real implementations or lightweight fakes for internal services
import { describe, it, expect, vi } from 'vitest';
import { createOrder } from './orderService';
import { db } from './database'; // Real test database (or in-memory SQLite)
import { inventoryService } from './inventoryService'; // Real implementation
import * as paymentService from './paymentService';

// Only mock the external payment gateway to avoid real charges
vi.spyOn(paymentService, 'chargeCard').mockResolvedValue({
  success: true,
  transactionId: 'tx_test_001'
});

describe('createOrder', () => {
  it('creates order and decrements inventory on success', async () => {
    await db.seed({ products: [{ id: 'p1', stock: 5, price: 29.99 }] });

    const result = await createOrder({ items: [{ productId: 'p1', qty: 2 }], userId: 'u1' });

    expect(result.status).toBe('confirmed');
    expect(await inventoryService.getStock('p1')).toBe(3); // Real decrement occurred
  });

  it('rolls back if payment fails', async () => {
    vi.spyOn(paymentService, 'chargeCard').mockRejectedValue(new Error('Card declined'));
    await db.seed({ products: [{ id: 'p1', stock: 5, price: 29.99 }] });

    await expect(
      createOrder({ items: [{ productId: 'p1', qty: 2 }], userId: 'u1' })
    ).rejects.toThrow('Card declined');

    expect(await inventoryService.getStock('p1')).toBe(5); // Stock not decremented
  });
});

The Happy-Path-Only Anti-Pattern (TST-005)

// ❌ BAD - Only the success case is tested
describe('getUserById', () => {
  it('returns a user when found', async () => {
    const user = await getUserById('user-123');
    expect(user.name).toBe('Alice');
  });
  // Missing: user not found (404), database error, invalid ID format,
  // deleted user, suspended user, insufficient permissions
});

// ✅ GOOD - Full range of cases including error paths
describe('getUserById', () => {
  it('returns user data for a valid existing user', async () => {
    const user = await getUserById('user-123');
    expect(user).toMatchObject({ id: 'user-123', name: 'Alice', status: 'active' });
  });

  it('throws UserNotFoundError when ID does not exist', async () => {
    await expect(getUserById('nonexistent-id')).rejects.toThrow(UserNotFoundError);
  });

  it('throws ValidationError for malformed IDs', async () => {
    await expect(getUserById('')).rejects.toThrow('User ID cannot be empty');
    await expect(getUserById(null as any)).rejects.toThrow(ValidationError);
  });

  it('returns null for soft-deleted users', async () => {
    const result = await getUserById('deleted-user-456');
    expect(result).toBeNull();
  });

  it('throws PermissionError when caller lacks read access', async () => {
    await expect(
      getUserById('private-user-789', { callerId: 'other-user' })
    ).rejects.toThrow(PermissionError);
  });
});

The Testing Anti-Pattern Comparison

Assertion-Free Tests: The Silent Failure (TST-008)

A third pattern that AI tools produce is the assertion-free test - a test function that calls some code but never asserts anything about the result. These tests always pass because there is no condition that could cause them to fail:

// ❌ BAD - No assertions: this test always passes regardless of behaviour
it('processes the payment', async () => {
  const result = await processPayment({ amount: 100, currency: 'USD' });
  console.log(result); // Logging is not asserting
});

// ✅ GOOD - Meaningful assertions about actual outcomes
it('processes the payment and returns a transaction ID', async () => {
  const result = await processPayment({ amount: 100, currency: 'USD' });
  expect(result.success).toBe(true);
  expect(result.transactionId).toMatch(/^tx_/);
  expect(result.amount).toBe(100);
});

Vitest and Jest both support an assertion count guard: calling expect.hasAssertions() at the start of a test causes it to fail if no assertions ran. This catches both assertion-free tests and tests where the assertion is inside an async callback that never executed.

How to Improve Your AI-Generated Test Suite

1. Audit your test files for over-mocking: Count the vi.mock() and jest.mock() calls in each test file. If you're mocking more than 2-3 dependencies, ask whether each mock is truly necessary or whether a lighter-weight approach (spy, partial mock, or real implementation) would provide better coverage.
2. Add error-path tests for every happy-path test: For each test that verifies success behaviour, write at least one test that verifies error behaviour. What happens when the database is unavailable? When the user doesn't exist? When the input is invalid?
3. Use expect.hasAssertions(): Add this call to the top of each test function to guarantee that assertion-free tests fail rather than silently pass.
4. Run automated scanning: Tools like VibeDoctor (vibedoctor.io) automatically scan your codebase for over-mocked tests, happy-path-only patterns, and assertion-free test blocks (TST-004, TST-005, TST-008) and flag specific file paths and line numbers. Free to sign up.
5. Add mutation testing: Tools like Stryker can automatically introduce small bugs into your code and check whether your tests catch them. If a test suite fails to catch mutations, it's a strong signal that coverage is superficial.

FAQ