Principle:Apache Spark Programmatic Application Launch
Metadata
| Field | Value |
|---|---|
| Domains | API_Design, Deployment |
Overview
A programmatic API pattern that enables launching and managing distributed applications as child processes from within JVM applications using a builder interface.
Description
While CLI tools like spark-submit serve interactive use, production systems often need to launch Spark applications programmatically from orchestration frameworks, web services, or scheduling systems. The programmatic launch pattern wraps the CLI submission process behind a type-safe Builder API that constructs the command, spawns the process, and provides a handle for lifecycle monitoring. This enables integration with enterprise workflow engines without shell scripting.
Key motivations for programmatic launch:
- Orchestration integration — workflow engines (Airflow, Oozie, custom schedulers) can submit and track Spark jobs through native JVM APIs
- Dynamic configuration — submission parameters can be computed at runtime based on data volume, cluster state, or business rules
- Error handling — JVM exception semantics replace shell exit codes for more robust failure handling
- Lifecycle management — the returned handle enables monitoring, stopping, and killing applications through a structured API
The pattern separates configuration (builder phase) from execution (launch phase) from monitoring (handle phase), following the principle of single responsibility.
Usage
Use this when building systems that need to submit Spark applications programmatically — such as job schedulers, web dashboards, or workflow orchestrators.
Theoretical Basis
Implements the Builder Pattern for configuration followed by the Process Handle pattern for lifecycle management. The handle provides an observable state machine:
UNKNOWN -> CONNECTED -> SUBMITTED -> RUNNING -> FINISHED | FAILED | KILLED | LOST
The Builder Pattern ensures:
- Type safety — configuration errors are caught at compile time rather than at submission time
- Fluent API — method chaining provides readable configuration code
- Separation of construction and representation — the same builder can produce different internal command structures
The Process Handle pattern provides:
- Abstraction over OS processes — the handle hides platform-specific process management details
- Observable state — listeners can react to state changes asynchronously
- Control operations — stop, kill, and disconnect operations are available through the handle