Workflow:DataExpert io Data engineer handbook PySpark Job Testing
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, Apache_Spark, Testing |
| Last Updated | 2026-02-09 06:00 GMT |
Overview
End-to-end process for unit testing PySpark transformation jobs using pytest fixtures and the chispa DataFrame comparison library.
Description
This workflow demonstrates how to test PySpark data transformations in isolation, without requiring a running Spark cluster or real data sources. It uses pytest with a session-scoped SparkSession fixture, constructs test DataFrames from named tuples, runs transformation functions against them, and validates results using chispa's assert_df_equality. This pattern ensures that data engineering logic is correct before deploying to production environments.
Usage
Execute this workflow when you have written or modified PySpark transformation functions and need to verify their correctness. This is appropriate after implementing the transformation jobs from the Spark Fundamentals module and before submitting homework or deploying transformations to a Spark cluster.
Execution Steps
Step 1: Install_Dependencies
Install the Python packages required for testing PySpark jobs locally. The requirements file specifies pyspark, pytest, and chispa as the core dependencies. Spark must also be installed and configured locally on the development machine.
Key considerations:
- requirements.txt pins: pyspark, pytest, chispa, and pyspark-test
- Local Spark installation is required (not Dockerized for testing)
- chispa provides DataFrame equality assertions that produce readable diff output
- Python environment should match the Spark cluster's Python version
Step 2: Configure_SparkSession_Fixture
Set up a pytest fixture that provides a shared SparkSession across all tests in the session. The fixture is defined in conftest.py with session scope, meaning a single SparkSession is created once and reused by all test functions, avoiding the overhead of repeated session creation.
What happens:
- conftest.py defines a session-scoped pytest fixture named "spark"
- SparkSession.builder creates a local-mode session with appName "chispa"
- The fixture is automatically injected into any test function that declares "spark" as a parameter
- Session scope means one SparkSession serves all tests in the test run
Step 3: Define_Test_Data
Create test input data using Python named tuples that mirror the schema of the source tables. Named tuples provide a readable, self-documenting way to construct test rows. Multiple test cases should cover normal operation, edge cases (empty arrays, null values), and partition filtering.
Pattern:
- Define named tuples matching source table columns
- Create input data covering: basic case, empty/null edge cases, partition boundary cases
- Use spark.createDataFrame(input_data) to build test DataFrames
Step 4: Execute_Transformation_Under_Test
Call the transformation function being tested, passing the SparkSession fixture and the test DataFrame. The transformation function registers the DataFrame as a temporary view and executes its SQL logic, returning a result DataFrame.
What happens:
- The transformation function (e.g., do_player_scd_transformation) receives spark and input DataFrame
- It registers the input as a temporary SQL view
- It executes the SQL query and returns the transformed DataFrame
- No external data sources are needed; the test is fully self-contained
Step 5: Assert_Results
Compare the actual output DataFrame against an expected DataFrame using chispa's assert_df_equality function. Define expected values using named tuples that match the output schema, create an expected DataFrame, and assert equality. If the DataFrames differ, chispa produces a detailed diff showing mismatched rows and columns.
Key considerations:
- assert_df_equality compares schema, data types, row count, and values
- Expected DataFrames are built from named tuples matching the output schema
- Test failures show readable diffs indicating exactly which rows or columns differ
- Run tests with python -m pytest from the module root directory