How We Reduced Test Flakiness by 94%

Flaky tests are the bane of every engineering team. Here's how we eliminated them from our own test suite at Qovr.

The State of Our Tests

In early 2025, our internal test suite had a 23% flakiness rate. That means nearly 1 in 4 test runs failed for reasons unrelated to actual code changes. This was costing us:

Root Cause Analysis

We categorized every flaky failure over two weeks:

CausePercentage

Timing/Race conditions42% Network instability28% Test isolation issues18% Resource constraints12%

The Fixes

Timing Issues (42%)

We replaced all sleep() calls with proper waiting strategies:

// Before
await sleep(2000);
await click('#submit');
// After
await waitForNetworkIdle();
await waitForElement('#submit', { state: 'visible' });
await click('#submit');

Network Instability (28%)

We implemented automatic retry with exponential backoff for all API calls:

async function fetchWithRetry(url, options, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fetch(url, options);
    } catch (e) {
      if (i === maxRetries - 1) throw e;
      await sleep(Math.pow(2, i) * 1000);
    }
  }
}

Test Isolation (18%)

We ensured each test creates its own data:

// Before - shared user
const testUser = 'test@example.com';
// After - unique user per test
const testUser = test-${Date.now()}@example.com;

Resource Constraints (12%)

We reduced parallelism during peak CI times and implemented better cleanup.

Results

After implementing these changes:

Lessons Learned

The path to reliable tests is paved with data. Start measuring, and the fixes will become obvious.