Implementation:Recommenders team Recommenders Databricks Install

Knowledge Sources	Recommenders
Domains	Deployment, Databricks, Cluster Management, Azure
Last Updated	2026-02-10 00:00 GMT

Overview

The databricks_install.py script automates the installation of the recommenders package and its dependencies onto a Databricks cluster, with optional support for MMLSpark and operationalization setup.

Description

This command-line tool uses the databricks-cli library to interact with a Databricks workspace via its REST API. The main workflow proceeds through several stages:

1. Authentication -- Connects to the Databricks workspace using a CLI profile via ProfileConfigProvider and _get_api_client.
2. Cluster Setup -- Optionally creates a new cluster with DEFAULT_CLUSTER_CONFIG (Standard_D3_v2 nodes, 2-8 workers, auto-termination at 120 minutes). Verifies the target cluster is RUNNING, polling for up to 5 minutes if in PENDING state.
3. Prerequisites -- Installs PyPI prerequisites (pip==21.2.4, setuptools==54.0.0, numpy==1.18.0) onto the cluster.
4. Recommenders Package -- Installs the recommenders package from PyPI and polls until installation completes or fails.
5. Additional Dependencies -- Installs Azure SDK dependencies (azure-cli-core, azure-mgmt-cosmosdb, azureml-sdk, azure-storage-blob).
6. MMLSpark (optional) -- Installs MMLSpark/SynapseML as a Maven library if the --mmlspark flag is set.
7. Operationalization (optional) -- The prepare_for_operationalization function downloads a version-specific CosmosDB Spark connector JAR (supporting Spark versions 3-9), uploads it to DBFS, and installs additional PyPI packages (pydocumentdb).
8. Restart -- Restarts the cluster to apply all installations.

Helper functions include dbfs_file_exists for checking DBFS file existence and get_installed_libraries for querying installed library statuses.

Usage

Use this script to automate Databricks cluster setup for running recommender system experiments at scale. It replaces a multi-step manual process of configuring cluster libraries, handling version-specific dependencies, and preparing for model operationalization with Azure services. Requires databricks-cli to be installed and authenticated.

Code Reference

Source Location

• Repository: Recommenders
• File: tools/databricks_install.py
• Lines: 1-354

Signature

def dbfs_file_exists(api_client, dbfs_path)
def get_installed_libraries(api_client, cluster_id)
def prepare_for_operationalization(cluster_id, api_client, dbfs_path, overwrite, spark_version)

# Command-line entry point
if __name__ == "__main__":
    # argparse-based CLI with arguments:
    #   --profile, --path-to-recommenders, --dbfs-path, --overwrite,
    #   --prepare-o16n, --mmlspark, --create-cluster, cluster_id

Import

from tools.databricks_install import dbfs_file_exists, get_installed_libraries, prepare_for_operationalization

I/O Contract

Inputs

Name	Type	Required	Description
cluster_id	str	Yes	Databricks cluster ID (or cluster name if --create-cluster is used)
--profile	str	No	Databricks CLI profile name (default: "DEFAULT")
--path-to-recommenders	str	No	Path to the recommenders repository root (default: ".")
--dbfs-path	str	No	DBFS directory for uploading files (default: "dbfs:/FileStore/jars")
--overwrite	flag	No	Whether to overwrite existing files on DBFS
--prepare-o16n	flag	No	Whether to install additional libraries for operationalization
--mmlspark	flag	No	Whether to install MMLSpark/SynapseML
--create-cluster	flag	No	Whether to create a new cluster with default configuration
api_client	ApiClient	Yes	Databricks API client for workspace authentication (for helper functions)
dbfs_path	str	Yes	DBFS path to check or upload to (for helper functions)
spark_version	str	Yes	Spark version string for selecting the correct CosmosDB JAR (for operationalization)

Outputs

Name	Type	Description
dbfs_file_exists return	bool	True if the file exists on DBFS, False otherwise
get_installed_libraries return	dict	Dictionary mapping package names to installation status strings
prepare_for_operationalization return	list	List of dictionaries describing installed libraries (JARs and PyPI packages)

Usage Examples

Basic Usage

# From the command line:
# Install recommenders onto an existing cluster
python tools/databricks_install.py my-cluster-id

# Create a new cluster and install with MMLSpark
python tools/databricks_install.py my-cluster-name \
    --create-cluster \
    --mmlspark \
    --profile MY_PROFILE

# Full setup with operationalization
python tools/databricks_install.py my-cluster-id \
    --prepare-o16n \
    --mmlspark \
    --overwrite \
    --dbfs-path "dbfs:/FileStore/jars"