Environment:Pytorch Serve Python PyTorch Runtime

Knowledge Sources	Pytorch_Serve PyTorch
Domains	Infrastructure, Deep_Learning
Last Updated	2026-02-13 00:00 GMT

Overview

Core Python 3.8+, PyTorch, and Java runtime environment required for all TorchServe operations.

Description

This environment defines the base software stack for running TorchServe. It requires a Python runtime (3.8 or later), a compatible PyTorch installation, and a Java runtime (for the TorchServe frontend/server process). The Python layer handles model inference via custom handlers, while the Java frontend manages HTTP/gRPC endpoints, worker lifecycle, and request routing. Additional Python packages (psutil, packaging, pynvml, etc.) are required for system metrics, version detection, and GPU monitoring.

Usage

Use this environment for all TorchServe deployments, whether CPU-only or GPU-accelerated. It is the mandatory base prerequisite for every TorchServe workflow: Model Deployment, LLM Deployment with vLLM, Large Model Inference, and HuggingFace Transformer Serving.

System Requirements

Category	Requirement	Notes
OS	Linux (Ubuntu 20.04+), macOS, or Windows	Linux recommended for production
Hardware	CPU (any x86_64 or aarch64)	GPU optional, see CUDA_GPU_Environment
Disk	5GB minimum	More for model artifacts and dependencies
Java	JDK 17+	Required for TorchServe frontend process

Dependencies

System Packages

Java Development Kit (JDK) 17+
`git` (for model downloads)

Python Packages

`torch` (PyTorch) — version depends on use case:
- >= 1.8.1 for profiler support
- >= 2.0.0 for `torch.compile` support
- > 2.2.2 for `torch._export.aot_compile` support
`psutil` == 5.9.8
`requests` == 2.32.0
`packaging` == 23.2
`pynvml` == 11.5.0
`pyyaml` == 6.0.1
`ninja` == 1.11.1.1
`setuptools`
`captum` == 0.6.0 (for model explainability)
`Pillow` (for image handlers)

Credentials

The following environment variables may be set for TorchServe configuration:

`JAVA_HOME`: Path to Java installation directory. If not set, TorchServe looks for `java` on PATH.
`TS_CONFIG_FILE`: Path to TorchServe configuration file (`config.properties`).
`TEMP`: Custom temporary directory for Java (Windows).
`ENABLE_TORCH_PROFILER`: Set to `TRUE` to enable PyTorch profiler output.
`TS_BENCHMARK`: Set to `true` to enable benchmarking mode in workers.

Quick Install

# Install TorchServe and model archiver
pip install torchserve torch-model-archiver torch-workflow-archiver

# Or install from source with dependencies
python ts_scripts/install_dependencies.py
pip install -e .

Code Evidence

Java requirement from `ts/model_server.py:75-76`:

java_home = os.environ.get("JAVA_HOME")
java = "java" if not java_home else "{}/bin/java".format(java_home)

PyTorch version checks from `ts/torch_handler/base_handler.py:23-28`:

if packaging.version.parse(torch.__version__) >= packaging.version.parse("1.8.1"):
    from torch.profiler import ProfilerActivity, profile, record_function
    PROFILER_AVAILABLE = True
else:
    PROFILER_AVAILABLE = False

PyTorch 2.0 compile detection from `ts/torch_handler/base_handler.py:42-54`:

if packaging.version.parse(torch.__version__) >= packaging.version.parse("2.0.0a"):
    PT2_AVAILABLE = True
else:
    logger.warning(
        f"Your torch version is {torch.__version__} which does not support torch.compile"
    )
    PT2_AVAILABLE = False

Platform-specific path handling from `ts/model_server.py:110-114`:

platform_path_separator = {"Windows": "", "Darwin": ".:", "Linux": ".:"}
class_path = "{}{}".format(
    platform_path_separator[platform.system()],
    os.path.join(ts_home, "ts", "frontend", "*"),
)

Common Errors

Error Message	Cause	Solution
`JAVA_HOME is not set` / Java not found	Java not installed or JAVA_HOME not configured	Install JDK 17+ and set `JAVA_HOME` environment variable
`Your torch version ... does not support torch.compile`	PyTorch version < 2.0	Upgrade to PyTorch >= 2.0 for torch.compile support
`No model weights could be loaded`	Model file format not recognized	Ensure model file is `.pt`, `.onnx`, or `.so` format
`TorchServe is already running`	Previous instance still active	Run `torchserve --stop` first, then restart

Compatibility Notes

Windows: Supported but less tested. Use WSL2 for best experience. Path separators differ from Linux/macOS.
macOS (Darwin): CUDA is not supported on macOS. Use CPU or Apple MPS (Metal Performance Shaders) for GPU acceleration.
Linux aarch64 (ARM64): Experimental support. Architecture-specific PyTorch wheels required.
Python versions: Tested with Python 3.8, 3.9, 3.10, 3.11 in Docker builds.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment