Principle:MarketSquare Robotframework browser Test Execution

Overview

The robotframework-browser project employs a multi-level testing strategy that combines unit tests (via pytest) and acceptance tests (via Robot Framework). The testing infrastructure supports parallel execution, test filtering, sharding for CI, and approval-based snapshot testing. Both test levels are orchestrated through Invoke tasks.

Core Concept

Testing a polyglot library that bridges Python and Node.js requires testing at multiple levels:

• Unit tests (utest) -- Test Python-side logic in isolation: argument parsing, type conversions, utility functions, and generated code
• Acceptance tests (atest) -- Test the full stack end-to-end: Python keywords calling Node.js code which drives Playwright to interact with web pages

The acceptance tests are the primary quality gate for the project, as they validate the complete integration between all layers.

Unit Tests (utest)

Framework

Unit tests use pytest as the test runner. The project uses several pytest plugins:

• pytest-approvaltests -- Enables approval testing (snapshot-based assertions) where test outputs are compared against pre-approved golden files
• JUnit XML output -- Test results are written in xunit format for CI integration

Approval Testing Pattern

Some unit tests use the approval testing pattern:

1. The test produces an output (e.g., a generated string or data structure)
2. The output is compared against a pre-approved "golden" file
3. If the output differs, the test fails and shows a diff
4. The developer reviews the diff and either fixes the code or approves the new output

This is particularly useful for testing generated code (like Python wrappers for JS keywords) where the exact output matters.

Test Organization

Unit tests are located in the utest/ directory. Test results are written to utest/output/ including:

• pytest_xunit.xml -- JUnit-format test results
• Log output with log_cli=True and log_cli_level=INFO

Acceptance Tests (atest)

Framework

Acceptance tests use Robot Framework as the test runner and pabot (parallel Robot Framework executor) for parallel execution. The test infrastructure includes:

• pabot -- Distributes test suites across multiple processes for faster execution
• robotstatuschecker -- Post-processes Robot Framework output to verify expected statuses
• rebot -- Merges parallel test outputs into a single report

Parallel Execution

By default, acceptance tests run in parallel using cpu_count - 1 processes (minimum 1). The --processes parameter allows overriding this. On GitPod environments, the process count is capped at 6.

The --ordering flag with atest/atest_order.data controls suite execution order for optimal parallel distribution.

Test Filtering

Tests can be filtered by multiple criteria:

Filter	Flag	Description
Suite	--suite	Run only a specific test suite
Test name	--test	Run only tests matching a name pattern
Tag include	--include	Run only tests with a specific tag
Tag exclude	--exclude	Skip tests with a specific tag
Shard	--shard	Run only a specific shard of tests (for CI parallelism)
Smoke	--smoke	Exclude tests tagged as "slow"

Platform-Specific Exclusions

Tests are automatically excluded based on the running platform:

• On Windows: tests tagged no-windows-support are excluded
• On macOS: tests tagged no-mac-support are excluded (unless --include-mac is set)
• Tests tagged tidy-transformer are always excluded
• Tests tagged not-implemented are always excluded

Test Application

Acceptance tests require a running test application (built via inv create-test-app). The test infrastructure spawns a Node.js process that serves the test application and configures the environment variable ROBOT_FRAMEWORK_BROWSER_NODE_PORT.

Iframe Testing

The --framed option runs tests in an iframe context by setting:

• SUFFIX:framing.html?url= -- Routes page loads through an iframe wrapper
• SELECTOR_PREFIX:id=iframe_id >>> -- Prefixes all selectors with iframe targeting
• Tests tagged no-iframe are excluded in this mode

Domains

• Testing -- Both unit and acceptance testing methodologies
• Quality_Assurance -- The testing strategy ensures correctness across the full stack

Implemented By

• Implementation:MarketSquare_Robotframework_browser_Inv_Utest_Atest

Related Topics

• MarketSquare_Robotframework_browser_Inv_Utest_Atest -- Implementation details of the test execution tasks
• MarketSquare_Robotframework_browser_Project_Build_Pipeline -- Building before testing
• MarketSquare_Robotframework_browser_Code_Quality_Enforcement -- Linting and type checking complement testing