Implementation:MarketSquare Robotframework browser Inv Utest Atest

Document Type

API Doc -- Documents the Invoke tasks for running unit tests (utest) and acceptance tests (atest) in the robotframework-browser project.

API Summary

Two primary test tasks are provided:

• utest -- Runs pytest-based unit tests with optional approval test reporter and suite selection
• atest -- Runs Robot Framework acceptance tests in parallel via pabot with extensive filtering options

Source References

File	Lines	Description
tasks.py	L343-372	utest task
tasks.py	L397-467	atest task
tasks.py	L604-645	_run_pabot helper
tasks.py	L648-659	_add_skips helper

CLI Usage

# Run all unit tests
inv utest

# Run a specific unit test suite
inv utest --suite utest/test_conversion.py

# Run all acceptance tests
inv atest

# Run a specific acceptance test suite
inv atest --suite "05_JS_Tests"

# Run a specific test by name
inv atest --test "My Funky Keyword"

# Run only fast tests (exclude slow)
inv atest --smoke

# Run with specific number of parallel processes
inv atest --processes 4

# Run a specific shard for CI
inv atest --shard "1/3"

Task: utest

Signature

@task
def utest(c, reporter=None, suite=None):
    """Run utest."""

Parameters

Parameter	Type	Default	Description
reporter	str or None	None	Full path to a diff program for approval test reporter (see pytest-approvaltests)
suite	str or None	None	Path to a specific test suite file to run

Source Code

@task
def utest(c, reporter=None, suite=None):
    """Run utest.

    Args:
        reporter: Defines which approval test reporter to use.
                  Must be full path to the diff program.
        suite:    Defines which test suite file to run. Same as: pytest path/to/test.py
                  Must be path to the test suite file

    To create coverage use: coverage run -m invoke utest
    """
    args = [
        "--showlocals",
        "--junitxml=utest/output/pytest_xunit.xml",
        "--tb=long",
        "-o",
        "log_cli=True",
        "-o",
        "log_cli_level=INFO",
    ]
    if reporter:
        args.append(f"--approvaltests-add-reporter={reporter}")
    if suite:
        args.append(suite)
    status = pytest.main(args)
    raise Exit(status)

Behavior

1. Constructs pytest argument list with:
   • --showlocals -- Shows local variables in tracebacks
   • --junitxml=utest/output/pytest_xunit.xml -- Writes JUnit XML results
   • --tb=long -- Uses long traceback format
   • log_cli=True and log_cli_level=INFO -- Enables live log output
2. Optionally adds the approval test reporter
3. Optionally restricts to a specific test suite
4. Calls pytest.main(args) directly (in-process)
5. Raises Exit(status) with the pytest exit code

Coverage Support

To measure code coverage, run:

coverage run -m invoke utest

Task: atest

Signature

@task(clean_atest, create_test_app)
def atest(
    c,
    suite=None,
    test=None,
    include=None,
    shard=None,
    debug=False,
    include_mac=False,
    smoke=False,
    processes=None,
    framed=False,
    exclude=None,
    loglevel=None,
    batteries=False,
):
    """Runs Robot Framework acceptance tests with pabot."""

Dependencies

clean_atest (cleans previous test output) and create_test_app (builds the test application).

Parameters

Parameter	Type	Default	Description
suite	str or None	None	Select which suite to run
test	str or None	None	Select which test to run
include	str or None	None	Select tests by tag
shard	str or None	None	Shard tests (e.g., "1/3" for first of 3 shards)
debug	bool	False	Use robotframework-debugger as test listener
include_mac	bool	False	Do not exclude no-mac-support tags
smoke	bool	False	Exclude tests tagged as "slow"
processes	str or None	None	Number of parallel processes (default: cpu_count - 1)
framed	bool	False	Run tests in iframe context
exclude	str or None	None	Exclude tests by tag
loglevel	str or None	None	Set Robot Framework log level (default: DEBUG)
batteries	bool	False	Run tests with BrowserBatteries

Behavior

1. Process count: On GitPod, caps processes at 6
2. Argument construction: Builds pabot arguments with ordering file, pythonpath, and optional filters
3. Filter application:
   • --suite, --test, --include, --exclude are passed through to Robot Framework
   • --smoke adds --exclude slow
   • --debug adds --listener Debugger
   • --framed sets iframe variables and excludes no-iframe tests
4. Output directory: Creates atest/output/
5. Node.js process: Spawns a background Node.js process for the Playwright gRPC server
6. Test execution: Calls _run_pabot to execute tests
7. Cleanup: Kills the background Node.js process in a finally block
8. Exit: Calls sys.exit(rc) with the test return code

The _run_pabot Helper

def _run_pabot(extra_args=None, shard=None, include_mac=False, loglevel="DEBUG"):
    os.environ["ROBOT_SYSLOG_FILE"] = str(ATEST_OUTPUT / "syslog.txt")
    pabot_args = [
        sys.executable,
        "-m",
        "pabot.pabot",
        "--pabotlib",
        "--pabotlibport",
        "0",
        "--processes",
        EXECUTOR_COUNT,
        "--chunk",
        "--artifacts",
        "png,webm,zip",
        "--artifactsinsubfolders",
    ] + (["--shard", shard] if shard else [])
    # ... default_args with --xunit, --exclude, --loglevel, --outputdir ...
    process = subprocess.Popen(
        pabot_args + (extra_args or []) + default_args, env=os.environ
    )
    process.wait(ATEST_TIMEOUT)
    # Post-process output
    robotstatuschecker.process_output(output_xml)
    rc = rebot_cli(["--outputdir", str(ATEST_OUTPUT), output_xml], exit=False)
    return rc

Key aspects of pabot execution:

• --pabotlib -- Enables pabot's shared resource library
• --pabotlibport 0 -- Auto-assigns port for pabot communication
• --processes -- Number of parallel Robot Framework processes (default: cpu_count - 1)
• --chunk -- Groups test suites for efficient distribution
• --artifacts png,webm,zip -- Collects test artifacts (screenshots, videos, archives)
• ATEST_TIMEOUT -- 900 seconds (15 minutes) timeout for the entire test run

Platform-Specific Skips

def _add_skips(default_args, include_mac=False):
    if platform.platform().lower().startswith("windows"):
        default_args.extend(["--exclude", "no-windows-support"])
    if not include_mac and (
        platform.platform().lower().startswith("mac")
        or platform.platform().lower().startswith("darwin")
    ):
        default_args.extend(["--exclude", "no-mac-support"])
    default_args.extend(["--exclude", "tidy-transformer"])
    return default_args

Test Output

Output	Location	Description
Unit test results	utest/output/pytest_xunit.xml	JUnit XML format pytest results
Acceptance test output	atest/output/output.xml	Robot Framework XML output
Acceptance test log	atest/output/log.html	HTML log (generated by rebot)
Acceptance test report	atest/output/report.html	HTML report (generated by rebot)
System log	atest/output/syslog.txt	Robot Framework system log
Playwright log	atest/output/playwright-log.txt	Node.js Playwright wrapper log

Related

• MarketSquare_Robotframework_browser_Test_Execution -- Principle document for the testing strategy
• MarketSquare_Robotframework_browser_Inv_Build -- Building before testing
• MarketSquare_Robotframework_browser_Inv_Lint -- Code quality checks that complement testing

Requires Environment

• Environment:MarketSquare_Robotframework_browser_CI_GitHub_Actions

Uses Heuristic

• Heuristic:MarketSquare_Robotframework_browser_Shared_Node_Process_For_Parallel