Implementation:BerriAI Litellm CircleCI Config

Overview

Description

The .circleci/config.yml file is the primary CI/CD pipeline configuration for the LiteLLM project, running on CircleCI 2.1. At 4,661 lines, it defines a comprehensive testing, linting, building, and deployment pipeline that orchestrates over 50 distinct jobs. The configuration covers unit tests, integration tests, proxy tests, Docker image builds, UI builds, security scans, and package publishing for the entire LiteLLM ecosystem.

The pipeline targets Python 3.11 and 3.12 as primary runtimes, with additional Windows and Python 3.13 compatibility testing. It uses Docker-based executors for most jobs and includes database-dependent integration tests using PostgreSQL.

Usage

This file is consumed automatically by CircleCI when code is pushed to the repository. Jobs are triggered based on branch filter rules -- most jobs run only on the main branch and branches matching the pattern /litellm_.*/ (release branches). The publish_to_pypi job requires successful completion of nearly all other test jobs before it will execute.

# Branch filter pattern used throughout:
filters:
  branches:
    only:
      - main
      - /litellm_.*/

Data Schema

Top-Level Structure

Key	Type	Description
version	String	CircleCI config version (2.1)
orbs	Map	External orb dependencies (codecov, node, windows)
commands	Map	Reusable command definitions
jobs	Map	All job definitions (50+ jobs)
workflows	Map	Workflow orchestration with job ordering and filters

Orbs

Orb	Version	Purpose
codecov	codecov/codecov@4.0.1	Code coverage reporting
node	circleci/node@5.1.0	Node.js tooling for UI builds
win	circleci/windows@5.0	Windows executor support

Reusable Commands

Command	Description
setup_google_dns	Configures Google DNS (8.8.8.8, 8.8.4.4) alongside local DNS for reliable network access
setup_litellm_enterprise_pip	Installs the local enterprise/ package in editable mode
setup_litellm_test_deps	Full test dependency installation with caching; installs pytest, pydantic, mcp, and many other test libraries

Schema Fields

Key Jobs

Linting and Static Analysis

Job	Docker Image	Description
mypy_linting	cimg/python:3.12	MyPy type checking on the litellm/ directory
semgrep	cimg/python:3.12	Semgrep security scanning with custom rules from .semgrep/rules
check_code_and_doc_quality	cimg/python:3.12	Code and documentation quality checks

Core Unit Tests

Job	Parallelism	Description
local_testing_part1	4	Local tests A-M with coverage, split by timing
local_testing_part2	4	Local tests N-Z with coverage, split by timing
langfuse_logging_unit_tests	1	Langfuse logging-specific tests
caching_unit_tests	1	Cache and caching-related tests
auth_ui_unit_tests	1	Authentication UI tests

LLM Provider Tests

Job	Description
llm_translation_testing	LLM provider translation/transformation tests
realtime_translation_testing	Realtime/streaming translation tests
llm_responses_api_testing	OpenAI Responses API compatibility tests
google_generate_content_endpoint_testing	Google generate content endpoint tests
mcp_testing	Model Context Protocol tests
agent_testing	Agent-related tests
batches_testing	Batch API tests
image_gen_testing	Image generation tests
audio_testing	Audio transcription/speech tests
ocr_testing	OCR endpoint tests
search_testing	Search endpoint tests

Mapped Test Suites

Job	Description
litellm_mapped_tests_proxy_part1	Proxy mapped tests (part 1)
litellm_mapped_tests_proxy_part2	Proxy mapped tests (part 2)
litellm_mapped_tests_llms	LLM mapped tests
litellm_mapped_tests_core	Core library mapped tests
litellm_mapped_tests_mcps	MCP mapped tests
litellm_mapped_tests_integrations	Integration mapped tests
litellm_mapped_tests_litellm_core_utils	Core utils mapped tests
litellm_mapped_enterprise_tests	Enterprise feature mapped tests

Router and Proxy Tests

Job	Description
litellm_router_testing	Router integration tests
litellm_router_unit_testing	Router unit tests
litellm_proxy_unit_testing_key_generation	Proxy key generation tests
litellm_proxy_unit_testing_part1	Proxy unit tests (part 1)
litellm_proxy_unit_testing_part2	Proxy unit tests (part 2)
litellm_security_tests	Security-focused tests
litellm_assistants_api_testing	OpenAI Assistants API compatibility tests
pass_through_unit_testing	Pass-through endpoint tests
guardrails_testing	Guardrails system tests

Docker and E2E Tests

Job	Description
build_docker_database_image	Builds Docker image with PostgreSQL for E2E testing
build_and_test	Docker build and test
e2e_openai_endpoints	End-to-end OpenAI endpoint tests (requires Docker image)
e2e_ui_testing_chromium	Playwright UI tests on Chromium
e2e_ui_testing_firefox	Playwright UI tests on Firefox
proxy_logging_guardrails_model_info_tests	Proxy logging/guardrails E2E tests
proxy_spend_accuracy_tests	Spend tracking accuracy E2E tests
proxy_multi_instance_tests	Multi-instance proxy E2E tests
proxy_store_model_in_db_tests	Model DB storage E2E tests
proxy_build_from_pip_tests	Pip-based proxy build tests
proxy_pass_through_endpoint_tests	Pass-through endpoint E2E tests
db_migration_disable_update_check	Database migration tests
test_bad_database_url	Bad database URL handling tests

UI and Build Jobs

Job	Description
ui_build	Build the Next.js UI dashboard
ui_unit_tests	Run UI unit tests (requires ui_build)
helm_chart_testing	Helm chart validation tests

Compatibility Tests

Job	Description
using_litellm_on_windows	Windows compatibility (Python 3.11 via Chocolatey)
installing_litellm_on_python	Standard Python installation test
installing_litellm_on_python_3_13	Python 3.13 compatibility test

Publishing

Job	Description
publish_to_pypi	Publish package to PyPI (requires all tests to pass)
publish_proxy_extras	Publish proxy extras package
upload-coverage	Aggregate and upload code coverage to Codecov

Workflow Structure

The pipeline defines a single workflow named build_and_test that orchestrates all jobs:

Independent Jobs (run in parallel, no dependencies):

• All linting, unit test, and provider test jobs run concurrently
• Branch filters restrict execution to main and /litellm_.*/ branches

Dependent Jobs (require other jobs):

• ui_unit_tests requires ui_build
• e2e_ui_testing_* requires both ui_build and build_docker_database_image
• e2e_openai_endpoints requires build_docker_database_image
• proxy_*_tests require build_docker_database_image
• upload-coverage requires all test jobs
• publish_to_pypi requires virtually all jobs including linting, tests, E2E, and publish_proxy_extras

Usage Examples

How Jobs Use Test Splitting

The local testing jobs use CircleCI's test splitting feature for parallelism:

# From local_testing_part1 (4 parallel containers):
TEST_FILES=$(circleci tests glob "tests/local_testing/**/test_[a-mA-M]*.py")
echo "$TEST_FILES" | circleci tests run \
  --split-by=timings \
  --verbose \
  --command="xargs python -m pytest \
    -vv --cov=litellm --cov-report=xml \
    --junitxml=test-results/junit.xml \
    -n 4 --timeout=300 --timeout_method=thread"

How Coverage is Aggregated

Each test job persists its coverage files to a shared workspace, and the upload-coverage job collects them all:

# Each job renames and persists:
mv coverage.xml local_testing_part1_coverage.xml
# ...
- persist_to_workspace:
    root: .
    paths:
      - local_testing_part1_coverage.xml

How Publishing Gates Work

The publish_to_pypi job requires successful completion of over 40 jobs before it can execute, ensuring no broken code is released.

Related Pages

• Model Prices Database - Primary data file validated in CI tests
• Model Prices Backup - Backup file checked for consistency in CI