Implementation:Puppeteer Puppeteer Sort Test Expectations

Property	Value
sources	tools/sort-test-expectations.mjs
domains	Tools, Testing, CI
last_updated	2026-02-12 00:00 GMT

Overview

Description

The sort-test-expectations tool is a JavaScript script that sorts, cleans, and validates the test/TestExpectations.json file. This JSON file contains the expected outcomes (PASS, FAIL, SKIP, TIMEOUT) for Puppeteer's test suite across different configurations (browser, protocol, platform). The tool enforces consistent formatting and removes redundant or outdated entries.

The script performs the following operations:

1. Sorting -- Expectations are sorted by specificity (least specific first). Specificity is computed as the number of parameters plus a pattern score: wildcard-only patterns (*) score 0, partial wildcards score 1, and exact patterns score 2. Ties are broken by lexicographic comparison of the testIdPattern.
2. Normalization -- Within each expectation entry, the parameters, expectations, and platforms arrays are sorted alphabetically. Comments are removed from entries that only have a PASS expectation, as they are likely outdated.
3. Redundancy removal -- The script iterates backward through the sorted list and removes entries that are:
   • Covered by a less-specific entry with the same or broader expectations.
   • Have cdp and firefox parameters together (firefox-cdp is no longer tested).
   • Have only PASS expectations with no matching less-specific entry (since PASS is the default).
4. Pattern matching -- The testIdMatchesExpectationPattern function converts glob-style patterns (using * as wildcards) to regular expressions for matching test IDs.
5. Lint mode -- When invoked with --lint, the script checks that the committed file matches the expected sorted output and that all non-PASS entries have comments. It exits with code 1 on failures.
6. Write mode -- Without --lint, the script writes the cleaned and sorted expectations back to the file, formatted with Prettier.

Usage

This script is run as npm run format:expectations for formatting or with --lint in CI to validate the file.

Code Reference

Source Location

tools/sort-test-expectations.mjs

Signature

// Standalone ESM script
function testIdMatchesExpectationPattern(title: string, pattern: string): boolean;
function getSpecificity(item: TestExpectation): number;
function isSubset(superset: Set, subset: Set): boolean;

// Main script: sorts, cleans, and writes/validates TestExpectations.json

Import

// This is a standalone script, not a library module.
// Run: node tools/sort-test-expectations.mjs
// Lint: node tools/sort-test-expectations.mjs --lint

I/O Contract

Input	Type	Description
test/TestExpectations.json	JSON file	Array of test expectation objects
--lint (CLI flag)	Argument	When present, validates instead of writing
.prettierrc.cjs	Config file	Prettier configuration for output formatting

Output (write mode)	Type	Description
test/TestExpectations.json	JSON file	Sorted, cleaned, and Prettier-formatted expectations

Output (lint mode)	Type	Description
Exit code 0	Process exit	File is correctly formatted and all non-PASS entries have comments
Exit code 1	Process exit	File is not correctly formatted or entries lack comments

Expectation Entry	Field	Type	Description
--	testIdPattern	string	Glob pattern matching test IDs (* for wildcard)
--	parameters	string[]	Configuration parameters (e.g., 'chrome', 'cdp', 'webDriverBiDi')
--	expectations	string[]	Expected outcomes (PASS, FAIL, SKIP, TIMEOUT)
--	platforms	string[]	Target platforms (e.g., 'linux', 'darwin', 'win32')
--	comment	string	Optional comment explaining the expectation

Usage Examples

// Format the test expectations file
// npm run format:expectations
// -- or --
// node tools/sort-test-expectations.mjs

// Lint the test expectations file (CI)
// node tools/sort-test-expectations.mjs --lint

// Example TestExpectations.json entry:
// {
//   "testIdPattern": "[navigation.spec] navigation Page.goto should work",
//   "parameters": ["firefox", "webDriverBiDi"],
//   "expectations": ["FAIL"],
//   "platforms": ["linux", "darwin", "win32"],
//   "comment": "Firefox navigation issue #12345"
// }

Related Pages

• Environment:Puppeteer_Puppeteer_Node_18_Runtime