Environment:Deepset ai Haystack GPU Device Environment

Knowledge Sources	Haystack PyTorch Device
Domains	Infrastructure, Deep_Learning
Last Updated	2026-02-11 20:00 GMT

Overview

GPU-accelerated device environment supporting NVIDIA CUDA, Intel XPU, and Apple MPS backends with automatic device detection and fallback to CPU.

Description

This environment defines the hardware acceleration layer for Haystack components that run local ML models. Haystack implements a device abstraction layer (`ComponentDevice` and `Device`) that automatically detects available GPU hardware and selects the optimal backend. The detection follows a strict priority order: CUDA > XPU > MPS > CPU. Both MPS and XPU can be disabled via environment variables for debugging or compatibility purposes. If PyTorch is not installed, only CPU is available.

Usage

Use this environment when running components that perform local model inference such as SentenceTransformersDocumentEmbedder, SentenceTransformersTextEmbedder, TransformersSimilarityRanker, or ExtractiveReader. GPU acceleration is not required but significantly improves performance for embedding and inference operations.

System Requirements

Category	Requirement	Notes
OS	Linux (CUDA/XPU), macOS (MPS), Windows (CUDA)	Platform determines available backends
Hardware (CUDA)	NVIDIA GPU with CUDA support	Any modern NVIDIA GPU (GTX 10xx+)
Hardware (XPU)	Intel GPU with XPU support	Requires Intel oneAPI and PyTorch >= 2.6
Hardware (MPS)	Apple Silicon (M1/M2/M3)	macOS 12.3+ required
Software	PyTorch with appropriate backend	`torch.cuda`, `torch.xpu`, or `torch.backends.mps`

Dependencies

System Packages

CUDA: NVIDIA driver + CUDA toolkit (version matching PyTorch build)
XPU: Intel oneAPI toolkit
MPS: No additional system packages (built into macOS)

Python Packages

`torch` (with appropriate backend compiled in)

Credentials

No credentials required for GPU/device configuration.

Configuration Environment Variables:

`HAYSTACK_MPS_ENABLED`: Set to `"false"` to disable Apple MPS even when available (default: `"true"`)
`HAYSTACK_XPU_ENABLED`: Set to `"false"` to disable Intel XPU even when available (default: `"true"`)

Quick Install

# For NVIDIA CUDA
pip install torch --index-url https://download.pytorch.org/whl/cu121

# For CPU only
pip install torch --index-url https://download.pytorch.org/whl/cpu

# Disable MPS on macOS (if causing issues)
export HAYSTACK_MPS_ENABLED=false

# Disable XPU (if causing issues)
export HAYSTACK_XPU_ENABLED=false

Code Evidence

Device detection logic from `haystack/utils/device.py:493-528`:

def _get_default_device() -> Device:
    """
    Return the default device for Haystack.
    Precedence: GPU > XPU > MPS > CPU. If PyTorch is not installed, only CPU is available.
    """
    try:
        torch_import.check()

        has_mps = (
            hasattr(torch.backends, "mps")
            and torch.backends.mps.is_available()
            and os.getenv("HAYSTACK_MPS_ENABLED", "true") != "false"
        )
        has_cuda = torch.cuda.is_available()
        has_xpu = (
            hasattr(torch, "xpu")
            and hasattr(torch.xpu, "is_available")
            and torch.xpu.is_available()
            and os.getenv("HAYSTACK_XPU_ENABLED", "true") != "false"
        )
    except ImportError:
        has_mps = False
        has_cuda = False
        has_xpu = False

    if has_cuda:
        return Device.gpu()
    elif has_xpu:
        return Device.xpu()
    elif has_mps:
        return Device.mps()
    else:
        return Device.cpu()

Device type enum from `haystack/utils/device.py:19-24`:

class DeviceType(Enum):
    CPU = "cpu"
    GPU = "cuda"
    DISK = "disk"
    MPS = "mps"
    XPU = "xpu"

CI disabling of MPS/XPU from `.github/workflows/slow.yml:14-18`:

HAYSTACK_MPS_ENABLED: false
HAYSTACK_XPU_ENABLED: false

Common Errors

Error Message	Cause	Solution
`CUDA out of memory`	Insufficient GPU VRAM for model	Use a smaller model, reduce batch_size, or switch to CPU
`RuntimeError: MPS backend out of memory`	Apple Silicon memory pressure	Reduce batch_size or set `HAYSTACK_MPS_ENABLED=false`
`No CUDA GPUs are available`	CUDA driver not installed or GPU not detected	Install NVIDIA drivers and CUDA toolkit

Compatibility Notes

CUDA: Most widely tested backend; all CI tests run on CPU with CUDA/MPS/XPU disabled
MPS: Apple Silicon acceleration; can be disabled with `HAYSTACK_MPS_ENABLED=false` if unstable
XPU: Intel GPU support; requires PyTorch with XPU build; can be disabled with `HAYSTACK_XPU_ENABLED=false`
CPU fallback: Always available; used when PyTorch is not installed or no GPU detected
Device override: All model components accept a `device` parameter to explicitly set the device

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment