Principle:Triton inference server Server Signal Handling

Overview

Signal Handling is the principle governing how Triton Inference Server intercepts and responds to operating system signals to achieve graceful shutdown, crash diagnostics, and cross-platform process lifecycle management. The TritonSignal module registers handlers for both normal termination signals (SIGINT, SIGTERM) and error signals (SIGSEGV, SIGABRT), using a shared condition variable to coordinate with the main server thread. On Windows, equivalent console control handlers are registered. Error signals additionally produce stack traces via Boost.Stacktrace for post-mortem debugging.

Theoretical Basis

Why Signal Handling Matters for Inference Servers

Inference servers are long-running processes that manage substantial state: loaded models consuming GPU memory, active inference requests with allocated buffers, open network connections, shared memory mappings, and tracing state. An uncontrolled termination -- whether from kill -9, a container orchestrator's SIGTERM, or a user's Ctrl+C -- can leave GPU memory leaked, model files locked, shared memory segments orphaned, or in-flight responses lost. Proper signal handling ensures the server has an opportunity to:

Drain in-flight inference requests within a configurable timeout
Stop accepting new connections
Unload models and release GPU memory pools
Close network sockets and shared memory regions
Flush trace data to disk
Write final log entries

Signal Types and Handlers

On POSIX systems, the module registers two categories of signal handlers:

Signal	Category	Handler Behavior
`SIGINT`	Graceful termination	Set exit flag, notify condition variable
`SIGTERM`	Graceful termination	Set exit flag, notify condition variable
`SIGSEGV`	Error/crash	Print stack trace, restore default handler, re-raise signal
`SIGABRT`	Error/crash	Print stack trace, restore default handler, re-raise signal

Condition Variable Coordination

The signal handling module exports three global variables that the main server loop uses for coordination:

extern bool signal_exiting_;
extern std::mutex signal_exit_mu_;
extern std::condition_variable signal_exit_cv_;

The main thread waits on signal_exit_cv_ after starting all server endpoints. When a termination signal arrives, the handler sets signal_exiting_ to true under the mutex lock and calls signal_exit_cv_.notify_all(). This wakes the main thread, which then initiates the orderly shutdown sequence: stopping HTTP/gRPC/metrics servers, destroying the Triton server instance, and releasing resources.

Idempotent Signal Handling

The CommonSignalHandler function checks signal_exiting_ before setting it, ensuring that receiving multiple SIGTERM/SIGINT signals (a common occurrence during aggressive container shutdown) does not cause double-cleanup or race conditions. Only the first signal triggers the shutdown sequence; subsequent signals are no-ops.

Error Signal Diagnostics

For crash signals (SIGSEGV, SIGABRT), the handler uses Boost.Stacktrace (configured with addr2line for symbol resolution) to print a complete stack trace to stderr before the process terminates. After printing, the handler restores the default signal disposition with signal(signum, SIG_DFL) and re-raises the signal with raise(signum), ensuring the operating system generates a core dump file for detailed post-mortem analysis. This two-phase approach provides both immediate diagnostic output and a full core dump.

Windows Compatibility

On Windows, the module registers a Console Control Handler via SetConsoleCtrlHandler() that responds to:

CTRL_C_EVENT (Ctrl+C)
CTRL_CLOSE_EVENT (console window close)
CTRL_BREAK_EVENT (Ctrl+Break)
CTRL_LOGOFF_EVENT (user logoff)
CTRL_SHUTDOWN_EVENT (system shutdown)

All events route to the same CommonSignalHandler function, providing consistent behavior across platforms.

Integration with Server Exit Timeout

The signal handling mechanism works in conjunction with the exit_timeout_secs_ parameter (default 30 seconds) from the server configuration. When the main thread is woken by the condition variable, it passes this timeout to each server endpoint's Stop() method, which waits up to that duration for active connections to drain before forcefully terminating.

Signal Safety Considerations

The handler functions are designed to be as signal-safe as possible. The graceful shutdown handler only sets a boolean and signals a condition variable, avoiding malloc, I/O, or other non-signal-safe operations in the critical path. The error signal handler deliberately accepts the risk of calling Boost.Stacktrace (which is not strictly signal-safe) because at that point the process is already in a crashed state and diagnostic output is more valuable than strict compliance.

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment