Principle:MarketSquare Robotframework browser Project Build Pipeline

Overview

The robotframework-browser project is a polyglot project combining Python and TypeScript/Node.js code. Building the project requires a multi-stage pipeline that compiles protobuf definitions, builds Node.js code, creates test applications, and generates Python type stubs. The build pipeline is orchestrated using Invoke (Python task runner) with explicit task dependencies that ensure correct execution order.

Core Concept

A multi-stage build pipeline coordinates the compilation and generation steps required to produce a working distribution of the Browser library. The pipeline must handle:

Protobuf compilation -- Generating gRPC client/server stubs for both Python and Node.js from .proto definition files
TypeScript compilation and bundling -- Compiling the TypeScript Playwright wrapper and bundling it into a single JavaScript file via esbuild
Test application building -- Compiling the dynamic test application used by acceptance tests
Python type stub generation -- Creating .pyi files for IDE autocompletion and type checking
Dependency installation -- Ensuring both Python and Node.js dependencies are up to date

Task Dependency Chain

The build pipeline uses Invoke's task dependency system to ensure correct ordering:

build
  ├── deps            (install Python + Node.js dependencies)
  ├── protobuf        (compile .proto files for Python + Node.js)
  ├── node_build      (compile TypeScript, bundle JS, generate stubs)
  │     └── protobuf  (dependency: protobuf must run first)
  └── create_test_app (build the dynamic test application)

The top-level build task depends on deps, protobuf, node_build, and create_test_app. After all dependencies complete, build runs _gen_stub one final time to ensure type stubs are up to date.

Stage Details

Dependencies (deps)

Installs both Python and Node.js dependencies:

Python: Uses uv pip install with Browser/dev-requirements.txt and pyproject.toml
Node.js: Runs npm install and npx playwright install --with-deps
Supports incremental behavior: skips npm install if package-lock.json has not changed

Protobuf Compilation (protobuf)

Compiles .proto files from the protobuf/ directory into language-specific stubs:

Python stubs: Generated into Browser/generated/ using grpc_tools.protoc with --python_out, --grpc_python_out, and --mypy_out
Node.js stubs: Generated into node/playwright-wrapper/generated/ using grpc_tools_node_protoc with JavaScript and TypeScript outputs
Supports incremental behavior: skips if .proto files have not changed since last generation

Node Build (node_build)

Compiles and bundles the Node.js Playwright wrapper:

Runs npm run build which triggers esbuild to bundle TypeScript source into a single index.js
Copies static files from node/playwright-wrapper/static/ to Browser/wrapper/static/
Generates Python type stubs via _gen_stub

Test App (create_test_app)

Builds the dynamic test application used by acceptance tests:

Runs npm run build-test-app
Produces compiled output in node/dynamic-test-app/dist/

Stub Generation (_gen_stub)

Generates Python type stubs for IDE support:

Runs stubgen --output mypy_stub Browser to create raw stubs
Runs python -m Browser.gen_stub to process and refine the stubs into Browser/browser.pyi

Build Outputs

Output Path	Description
`Browser/generated/`	Python gRPC stubs (playwright_pb2.py, playwright_pb2_grpc.py, type stubs)
`node/playwright-wrapper/generated/`	Node.js gRPC stubs (JavaScript and TypeScript definitions)
`Browser/wrapper/index.js`	Bundled Node.js Playwright wrapper (single file distribution)
`Browser/wrapper/static/`	Static files served by the wrapper
`Browser/browser.pyi`	Python type stubs for IDE autocompletion
`node/dynamic-test-app/dist/`	Compiled test application

Incremental Build Support

The pipeline uses timestamp-based change detection to avoid unnecessary rebuilds:

_sources_changed(source_files, timestamp_file) compares modification times of source files against a timestamp marker file
If no sources have changed since the last build, the corresponding stage is skipped
The --force flag on individual tasks overrides this optimization

Domains

Build_Systems -- Invoke provides the task runner and dependency graph
Continuous_Integration -- The build pipeline is the foundation for CI/CD workflows

Implemented By

Implementation:MarketSquare_Robotframework_browser_Inv_Build

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment