Heuristic:Microsoft Playwright Test Stability Practices
| Knowledge Sources | |
|---|---|
| Domains | Testing, Optimization |
| Last Updated | 2026-02-11 22:00 GMT |
Overview
Test stability techniques including animation disabling for screenshots, browser download retry strategies, route error cleanup, and flaky test management via retries.
Description
Playwright incorporates several built-in stability mechanisms that test authors should understand and leverage. Screenshot comparisons default to disabling animations to prevent timing-dependent failures. Browser download logic retries up to 5 times with URL rotation. Route callbacks have a recommended cleanup pattern to prevent hanging tests. The test runner supports `--retries` for handling genuinely flaky tests and `--fail-on-flaky-tests` for strict CI enforcement.
Usage
Apply these practices when writing screenshot comparison tests, setting up CI/CD pipelines with browser installation, using network route handlers in tests, or diagnosing flaky test runs.
The Insight (Rule of Thumb)
- Action: Screenshot comparisons automatically set `animations: 'disabled'` to prevent flaky pixel differences from CSS animations and transitions.
- Action: When using `page.route()` callbacks, always clean up with `await page.unrouteAll({ behavior: 'ignoreErrors' })` in teardown to prevent hanging tests.
- Action: Use `--retries N` for flaky tests in CI, combined with `--fail-on-flaky-tests` when you want to detect flakiness without blocking the build.
- Action: Browser downloads retry 5 times automatically with URL rotation between configured download mirrors, so transient network errors are tolerated.
- Value: The registry installation lock uses exponential backoff over 10 minutes (20 retries, factor 1.27579), preventing race conditions when multiple CI jobs install browsers simultaneously.
- Trade-off: Disabling animations for screenshots means you cannot visually test animation correctness via snapshot comparison. Use dedicated animation tests with manual timing control instead.
- Trade-off: Retries mask genuine bugs. Track retry rates and investigate frequently-retried tests.
Reasoning
Animation-related flakiness is the most common cause of screenshot comparison failures. CSS transitions and animations produce different pixel output depending on when the screenshot is captured relative to the animation timeline. By pausing all animations and transitions, screenshots become deterministic. The route cleanup pattern exists because unresolved route callbacks keep the page in a pending state indefinitely, which manifests as timeout errors in subsequent tests. The browser download retry strategy addresses the reality that CDN downloads fail intermittently, especially in CI environments with rate limits or network instability.
Code Evidence
Animation disabling for screenshots from `packages/playwright/src/matchers/toMatchSnapshot.ts:346`:
// Screenshots default to animations: 'disabled'
Route error cleanup suggestion from `packages/playwright-core/src/client/network.ts:862-864`:
// Route callback error suggests using:
// await page.unrouteAll({ behavior: 'ignoreErrors' })
Browser download retry from `packages/playwright-core/src/server/registry/browserFetcher.ts:48-51`:
const retryCount = 5;
for (let attempt = 1; attempt <= retryCount; ++attempt) {
debugLogger.log('install', `downloading ${title} - attempt #${attempt}`);
const url = downloadURLs[(attempt - 1) % downloadURLs.length];
Registry lock exponential backoff from `packages/playwright-core/src/server/registry/index.ts:1061-1066`:
retries: {
// Retry 20 times during 10 minutes with
// exponential back-off.
retries: 20,
factor: 1.27579,
},
EBUSY error handling for mounted containers from `packages/playwright/src/runner/tasks.ts:220`:
// EBUSY error handling for Docker/mounted container environments
// See https://github.com/microsoft/playwright/issues/12106