Implementation:Apache Spark Assembly Plugin
Metadata
| Field | Value |
|---|---|
| Source Type | Doc |
| Source Name | Spark Submission Guide |
| Source URL | https://spark.apache.org/docs/latest/submitting-applications.html |
| Domains | Packaging |
| Type | External Tool Doc |
Overview
External tool documentation for sbt-assembly and Maven shade plugin used to create deployable Spark application JARs.
Description
For JVM applications, sbt-assembly (SBT) or maven-shade-plugin (Maven) creates uber JARs containing application code and all dependencies. For Python, the --py-files flag distributes .py, .zip, or .egg files. The --packages flag enables runtime Maven dependency resolution.
Key packaging approaches:
- sbt-assembly — SBT plugin that merges all dependency JARs into a single fat JAR, with merge strategy configuration for conflicting files
- maven-shade-plugin — Maven plugin that creates shaded JARs with optional package relocation to avoid classpath conflicts
- --py-files — spark-submit flag for distributing Python dependency files to executors
- --packages — spark-submit flag for runtime Maven coordinate resolution (downloads dependencies at submission time)
Usage
- Use sbt-assembly for SBT projects
- Use maven-shade-plugin for Maven projects
- Use --py-files for Python applications
Code Reference
Source: docs/submitting-applications.md (L25-39, L198-224)
Key CLI Flags
| Flag | Description |
|---|---|
| --jars | Comma-separated list of additional JARs to include on the driver and executor classpaths |
| --packages | Comma-separated list of Maven coordinates for runtime dependency resolution |
| --py-files | Comma-separated list of .py, .zip, or .egg files to distribute to Python executors |
| --repositories | Comma-separated list of additional Maven repositories to search |
I/O
Inputs
| Input | Type | Description |
|---|---|---|
| Application source + dependencies | build.sbt or pom.xml | Build definition file specifying project dependencies |
Outputs
| Output | Type | Description |
|---|---|---|
| Assembly/uber JAR | .jar file | Single JAR containing application and all dependencies (JVM) |
| Python archive | .zip or .egg file | Archive containing Python application dependencies |
Examples
SBT Assembly
sbt assembly
# Creates: target/scala-2.13/my-app-assembly.jar
Maven Shade
mvn package
# Creates: target/my-app-shaded.jar
Python with --py-files
spark-submit --py-files deps.zip app.py
Related
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment