Implementation:Evidentlyai Evidently CI Main Workflow

Knowledge Sources	Evidentlyai_Evidently
Domains	CI/CD, Testing, Build, Quality Assurance
Last Updated	2026-02-14 12:00 GMT

Overview

Defines the main CI/CD workflow for the Evidently repository, orchestrating changed file detection, linting, multi-platform Python testing, package building, API reference generation, UI building, UI code checking, UI service tests, and HTML visual regression tests.

Description

The main CI workflow is the primary continuous integration pipeline for the Evidently project. It is a comprehensive workflow that covers both the Python backend and the React UI frontend. The workflow uses a smart change detection strategy to only run relevant jobs based on which files were modified.

Triggers: The workflow runs on:

Push to main branch
Pull requests targeting main (opened, reopened, synchronized, ready_for_review)
Manual workflow dispatch

Concurrency: Uses cancel-in-progress: true grouped by workflow and ref, so new pushes cancel previous in-flight runs for the same branch.

Job Dependency Graph:

changed_files is the root job that determines which downstream jobs run. It categorizes changes into four groups:

evidently_python: Python source code, tests, config files
ui: UI source code
ui_build: UI build assets and nbextension
api_reference: API reference and GitHub action files

Jobs:

changed_files - Detects modified files using tj-actions/changed-files@v42 and sets output flags for conditional job execution. Skips draft PRs.

linter - Runs on Python changes. Installs PyTorch (CPU), minimal + dev dependencies, executes pre-commit run --all and mypy type checking. Uses Python 3.10.

prepare-cache-data - Caches test datasets (bikes dataset and scipy datasets) for downstream test jobs.

test-minimal - Tests with minimal requirements on Python 3.10/Ubuntu. Installs minimal dependencies and runs pytest with HTML report generation. Uploads test reports as artifacts.

test - Matrix test job across multiple platforms and Python versions. Matrix: Ubuntu 22.04, Windows 2022, macOS 14 with Python 3.10, 3.11, 3.12, 3.13 (excludes Windows + Python 3.13). Depends on linter passing. Installs with dev, spark, fsspec, llm, sql extras.

python-build - Builds the Python package using uv build and uploads the distribution as an artifact.

generate-api-reference - Generates API reference documentation when Python source or API reference files change. Uploads docs as artifacts.

ui-build - Builds the React UI using pnpm when UI files change.

ui-code-check - Runs TypeScript type checking and code quality checks on the UI when UI or Python files change (types are generated from the backend).

ui-test - Runs Playwright end-to-end tests against the UI service. Creates demo projects, starts the Evidently UI server, waits for readiness, and runs Playwright tests with Chromium. Uploads Playwright reports.

html-visual-testing - Runs Playwright visual regression tests on HTML report output. Converts Jupyter notebooks to HTML, serves them, and runs visual comparison tests. Uploads Playwright reports.

Usage

This workflow runs automatically on pushes and PRs to main. It can also be triggered manually via the GitHub Actions tab. The change detection ensures efficient CI runs by only executing relevant jobs. All test reports and build artifacts are uploaded for inspection with 5-30 day retention.

Code Reference

Source Location

Repository: Evidentlyai_Evidently
File: .github/workflows/main.yml

Signature

name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
    types: [opened, reopened, synchronize, ready_for_review]
  workflow_dispatch:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  changed_files:    # Detects which files changed
  linter:           # Pre-commit + mypy
  prepare-cache-data: # Cache test datasets
  test-minimal:     # Minimal dependency tests
  test:             # Multi-platform matrix tests
  python-build:     # Package build with uv
  generate-api-reference: # API docs generation
  ui-build:         # React UI build
  ui-code-check:    # TypeScript type/code check
  ui-test:          # Playwright UI service tests
  html-visual-testing: # Playwright visual regression

Import

This is a GitHub Actions workflow file and is not imported in Python code. It is triggered by GitHub events.

I/O Contract

Workflow Triggers

Trigger	Condition	Description
push	`branches: [main]`	Runs on every push to the main branch
pull_request	`branches: [main]`	Runs on PRs targeting main (non-draft only)
workflow_dispatch	(none)	Manual trigger from GitHub Actions tab

Changed File Groups

Group	Paths	Triggers Jobs
evidently_python	`src/evidently/`, `tests/`, `setup.py`, etc.	linter, test-minimal, test, python-build, html-visual-testing
ui	`ui/**`	ui-build, ui-code-check
ui_build	`src/evidently//ui//assets/`, `src/evidently/nbextension/`	ui-test
api_reference	`api-reference/`, `.github/`	generate-api-reference

Test Matrix

OS	Python Versions	Notes
ubuntu-22.04	3.10, 3.11, 3.12, 3.13	Full coverage
windows-2022	3.10, 3.11, 3.12	Python 3.13 excluded
macos-14	3.10, 3.11, 3.12, 3.13	ARM64 (Apple Silicon)

Artifacts Produced

Artifact Name	Source Job	Retention	Content
pytest-html-report	test-minimal	30 days	HTML test results
evidently-package-build-artifact	python-build	default	Python distribution files (wheel + sdist)
api-reference	generate-api-reference	5 days	Generated API documentation
ui-service-playwright-report	ui-test	30 days	Playwright test report
ui-html-visual-testing-playwright-report	html-visual-testing	30 days	Playwright visual regression report

Usage Examples

# The workflow is triggered automatically. To run manually:
# Go to Actions tab > CI > Run workflow

# To add a new Python test dependency, update:
# - requirements.min.txt (for minimal test job)
# - pyproject.toml (for matrix test job)

# To add a new file group for change detection, add to
# the changed_files job's files_yaml parameter:
#   my_new_group:
#     - path/to/files/**

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment