Principle:Apache Spark Dependency Bundling
Metadata
| Field | Value |
|---|---|
| Domains | Packaging, Deployment |
Overview
A packaging strategy that bundles application code with all its transitive dependencies into a single deployable artifact, eliminating classpath conflicts in distributed environments.
Description
In distributed computing, application code runs on multiple cluster nodes that may not have the same libraries installed. Dependency bundling solves this by creating uber/fat JARs that contain the application plus all dependencies (for JVM) or archive files (for Python). This ensures consistent execution across heterogeneous cluster environments.
The core challenge addressed by dependency bundling:
- Classpath isolation — each application carries its own dependency versions, avoiding conflicts with other applications or the cluster's system libraries
- Deployment atomicity — a single artifact contains everything needed to execute, reducing deployment failures from missing dependencies
- Version consistency — the exact same library versions used during development are guaranteed at runtime
- Environment independence — applications run identically regardless of what is installed on cluster nodes
For JVM applications (Scala/Java), the standard approach is creating an uber JAR using sbt-assembly or Maven shade plugin. For Python applications, dependencies are distributed via --py-files as .py, .zip, or .egg archives.
Usage
Use this before submitting any Spark application to a cluster. The packaging approach depends on the language:
- JVM apps — use sbt-assembly or Maven shade plugin to create uber JARs
- Python apps — use --py-files to distribute dependency archives
- Mixed apps — combine both approaches as needed
Theoretical Basis
Dependency resolution follows the Maven/Ivy dependency tree model with conflict resolution. The uber JAR creation merges all transitive dependencies into one archive, with shading to relocate conflicting packages.
The dependency tree resolution process:
- Transitive resolution — all dependencies of dependencies are recursively included
- Conflict mediation — when multiple versions of the same library exist, a deterministic strategy selects one (nearest-wins for Maven, latest-wins for Ivy)
- Shading/relocation — conflicting packages can be relocated to unique namespaces to allow multiple versions to coexist