Environment:Spotify Luigi Tornado Web Server

Knowledge Sources	Spotify Luigi Tornado
Domains	Infrastructure, Web_Server
Last Updated	2026-02-10 07:00 GMT

Overview

Tornado web server environment providing the HTTP server and IOLoop for the Luigi central scheduler daemon (`luigid`).

Description

This environment defines the Tornado web framework dependency that powers Luigi's central scheduler server. Tornado provides the HTTP server (`tornado.httpserver`), request handling (`tornado.web`), IO event loop (`tornado.ioloop`), and network utilities (`tornado.netutil`) used by `luigid`. The server exposes a REST API for worker-scheduler communication, serves the web visualiser UI, and provides task history endpoints.

Usage

This environment is required for running the central scheduler daemon (`luigid`). It is a mandatory prerequisite for the Central_Scheduler_Deployment workflow. Note that Tornado is a core dependency of Luigi and is always installed, but the server-specific functionality is only activated when running `luigid`.

System Requirements

Category	Requirement	Notes
OS	Linux, macOS, Windows	Cross-platform
Network	Available port for HTTP server	Default: 8082 (configurable)
Memory	Minimal	Depends on number of tracked tasks

Dependencies

Python Packages

`tornado` >= 5.0, < 7
`luigi` (core)

Credentials

No specific credentials required for the web server itself. Access control is managed externally (e.g., via firewall rules or reverse proxy).

Configuration in `luigi.cfg`:

`[scheduler] record_task_history`: Enable task history recording (default: False)
`[scheduler] state_path`: Path for scheduler state persistence

Quick Install

# Tornado is installed automatically with Luigi
pip install luigi

Code Evidence

Tornado imports from `luigi/server.py:48-51`:

import tornado.httpserver
import tornado.ioloop
import tornado.netutil
import tornado.web

Tornado web handlers from `luigi/server.py:93-299`:

class RPCHandler(tornado.web.RequestHandler):
    # ... handles scheduler RPC calls

class BaseTaskHistoryHandler(tornado.web.RequestHandler):
    # ... handles task history queries

class MetricsHandler(tornado.web.RequestHandler):
    # ... handles metrics endpoint

Server startup from `luigi/server.py:345-393`:

# Application setup
app = tornado.web.Application(...)

# HTTP server
http_server = tornado.httpserver.HTTPServer(app)

# Periodic scheduler state dumps
tornado.ioloop.PeriodicCallback(...)

# Start the IO loop
tornado.ioloop.IOLoop.current().start()

Common Errors

Error Message	Cause	Solution
`Address already in use`	Port 8082 (or configured port) is occupied	Use a different port or stop the conflicting process
`ImportError: No module named 'tornado'`	Tornado not installed	`pip install tornado` (should be automatic with Luigi)

Compatibility Notes

Tornado 5.x vs 6.x: Both major versions are supported. Tornado 6 uses native asyncio by default.
Tornado 7+: Not yet supported (upper bound is < 7).
UNIX sockets: The server supports both TCP/IP and UNIX socket modes for local-only access.
CORS: The server can be configured to accept cross-origin requests via configuration.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment