Implementation:Apache Spark SparkLauncher

Metadata

Field	Value
Source Type	Repo
Source Name	Apache Spark
Source URL	https://github.com/apache/spark
Domains	API_Design, Deployment
Type	API Doc

Overview

Primary Java API for programmatically launching Spark applications as child processes.

Description

SparkLauncher extends AbstractLauncher to provide a builder pattern for configuring and launching Spark applications. It supports two launch modes:

launch() — returns a raw Process handle for low-level process management
startApplication() — returns a SparkAppHandle with state tracking and listener support

The launcher constructs spark-submit commands internally using SparkSubmitCommandBuilder. This means the full power of spark-submit is available programmatically without shell scripting.

Usage

Use in Java/Scala applications that need to submit Spark jobs programmatically. Prefer startApplication() for production use as it provides lifecycle callbacks.

Code Reference

Source: Repository apache/spark, File launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java, Lines 41-506

Signature (Java)

public class SparkLauncher extends AbstractLauncher<SparkLauncher> {
    public SparkLauncher()
    public SparkLauncher(Map<String, String> env)
    public SparkLauncher setJavaHome(String javaHome)
    public SparkLauncher setSparkHome(String sparkHome)
    public Process launch() throws IOException
    public SparkAppHandle startApplication(Listener... listeners) throws IOException
}

Import

import org.apache.spark.launcher.SparkLauncher;

I/O

Inputs

Parameter	Type	Method	Description
master	String	setMaster()	Cluster manager URL
mainClass	String	setMainClass()	Application main class
appResource	String	setAppResource()	Path to application JAR or Python file
deployMode	String	setDeployMode()	client or cluster
conf key-value pairs	String, String	addSparkArg() / setConf()	Spark configuration properties

Outputs

Output	Type	Method	Description
Process	java.lang.Process	launch()	Raw OS process handle
SparkAppHandle	SparkAppHandle	startApplication()	High-level application handle with state tracking

Examples

Basic Launch

Process process = new SparkLauncher()
    .setMaster("spark://master:7077")
    .setMainClass("com.example.App")
    .setAppResource("/path/to/app.jar")
    .launch();

Launch with Listener

SparkAppHandle handle = new SparkLauncher()
    .setMaster("yarn")
    .setMainClass("com.example.App")
    .setAppResource("app.jar")
    .startApplication(new SparkAppHandle.Listener() {
        public void stateChanged(SparkAppHandle h) {
            System.out.println(h.getState());
        }
        public void infoChanged(SparkAppHandle h) {}
    });

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment