Principle:Snorkel team Snorkel Spark Mapper Adaptation
| Knowledge Sources | |
|---|---|
| Domains | Data_Processing, Distributed_Computing |
| Last Updated | 2026-02-14 20:38 GMT |
Overview
Technique for adapting mutable data-point transformation operations to work with immutable distributed data structures such as PySpark Row objects.
Description
In Snorkel's mapper infrastructure, data transformations operate by mutating fields on data point objects (e.g., SimpleNamespace or pandas.Series). However, distributed computing frameworks like PySpark use immutable data structures (Row objects) that cannot be modified in place. Spark Mapper Adaptation solves this impedance mismatch by replacing the field-update mechanism with one that reconstructs the data point from scratch, merging original and new fields into a fresh immutable object. This pattern is a specific instance of the broader adapter pattern applied to distributed data processing.
Usage
Apply this principle when extending Snorkel's mapper or preprocessor pipeline to operate over PySpark DataFrames. It is required whenever a Mapper that normally mutates data points needs to run in a Spark execution context where Row immutability is enforced.
Theoretical Basis
The core mechanism is an adapter pattern:
Pseudo-code Logic:
# Standard mapper: mutates data point in place
data_point.new_field = computed_value
# Spark-adapted mapper: reconstructs immutable Row
all_fields = original_row.asDict()
all_fields["new_field"] = computed_value
new_row = Row(**all_fields)
The adaptation replaces the internal _update_fields method on a Mapper instance. Since PySpark Row objects expose an asDict() method that returns a mutable dictionary representation, the adapted method:
- Extracts all existing fields into a dictionary
- Merges newly computed fields into that dictionary
- Constructs a new Row from the combined dictionary
This preserves the Mapper's external interface while changing only the field-update semantics.