Implementation:Spotify Luigi DropboxTarget

Knowledge Sources	Spotify_Luigi
Domains	Cloud_Storage, Dropbox
Last Updated	2026-02-10 08:00 GMT

Overview

Luigi contrib module providing Dropbox cloud storage integration through the DropboxClient filesystem and DropboxTarget target classes, with support for chunked uploads and OAuth2 authentication.

Description

The dropbox module implements Luigi's FileSystem and FileSystemTarget abstractions for the Dropbox cloud storage service. It depends on the dropbox Python package for API access.

Core Classes:

• DropboxClient (extends FileSystem): Provides authenticated filesystem operations against a Dropbox account. It requires an OAuth2 token for authentication and supports an optional root_namespace_id for interacting with Dropbox Team Spaces. The client implements exists, remove, mkdir, isdir, listdir, move, copy, download_as_bytes, and upload. Uploads use a chunked session-based approach (4MB chunks) via files_upload_session_start/files_upload_session_append_v2/files_upload_session_finish to handle large files.

• DropboxTarget (extends FileSystemTarget): Represents a file in Dropbox as a Luigi target. It requires a path (starting with '/') and an OAuth2 token. Supports reading via ReadableDropboxFile and writing via AtomicWritableDropboxFile. The temporary_path() context manager creates a local temporary file and uploads it to Dropbox upon successful completion.

• ReadableDropboxFile: File-like object that downloads blob content as bytes from Dropbox when read() is called.

• AtomicWritableDropboxFile (extends AtomicLocalFile): Writes to a local temporary file and uploads to Dropbox when the file is closed.

Decorators: The module includes two decorators for path normalization:

• accept_trailing_slash_in_existing_dirpaths: Strips trailing slashes from existing directory paths with a warning.
• accept_trailing_slash: Silently strips trailing slashes from all paths.

Usage

Use this module when your Luigi pipeline needs to read from or write to Dropbox. It requires creating a Dropbox API app and generating an OAuth2 access token. The module supports both "App folder" and "Full Dropbox" access levels.

Code Reference

Source Location

• Repository: Spotify_Luigi
• File: luigi/contrib/dropbox.py
• Lines: 1-331

Signature

class DropboxClient(FileSystem):
    def __init__(self, token, user_agent="Luigi", root_namespace_id=None):
        ...

class DropboxTarget(FileSystemTarget):
    def __init__(self, path, token, format=None, user_agent="Luigi", root_namespace_id=None):
        ...

Import

from luigi.contrib.dropbox import DropboxClient, DropboxTarget

I/O Contract

Inputs

Name	Type	Required	Description
path	str	Yes	Remote path in Dropbox, must start with '/' and should not end with '/'
token	str	Yes	A valid Dropbox OAuth2 access token
format	luigi.format.Format	No	Luigi format for encoding/decoding; defaults to get_default_format()
user_agent	str	No	User agent string for Dropbox API requests; defaults to "Luigi"
root_namespace_id	str	No	Root namespace ID for interacting with Dropbox Team Spaces

Outputs

Name	Type	Description
ReadableDropboxFile	file-like	Returned when opening target in read mode ('r'); downloads content as bytes
AtomicWritableDropboxFile	file-like	Returned when opening target in write mode ('w'); writes locally then uploads to Dropbox
bool (exists)	bool	exists() returns True if the file or folder exists in Dropbox

Usage Examples

Basic Usage

import luigi
from luigi.contrib.dropbox import DropboxTarget

class WriteToDropbox(luigi.Task):
    def output(self):
        return DropboxTarget(
            path='/my_app/output/results.csv',
            token='your-oauth2-token-here'
        )

    def run(self):
        with self.output().open('w') as f:
            f.write('id,name,value\n')
            f.write('1,Alice,100\n')
            f.write('2,Bob,200\n')

Reading from Dropbox

class ReadFromDropbox(luigi.Task):
    def requires(self):
        return WriteToDropbox()

    def run(self):
        with self.input().open('r') as f:
            content = f.read()
            # Process content...

    def output(self):
        return luigi.LocalTarget('/tmp/processed.txt')

Using temporary_path for Safe Writes

class SafeDropboxWrite(luigi.Task):
    def output(self):
        return DropboxTarget(
            path='/reports/daily_report.csv',
            token='your-oauth2-token-here'
        )

    def run(self):
        with self.output().temporary_path() as tmp_path:
            with open(tmp_path, 'w') as f:
                f.write('Generated report content\n')
            # File is uploaded to Dropbox when context manager exits successfully

Related Pages

• Principle:Spotify_Luigi_Cloud_Data_Targets