Principle:Triton inference server Server Memory Allocation Testing

Overview

Memory Allocation Testing is the principle of systematically verifying that Triton Inference Server correctly allocates, transfers, and validates data across CPU and GPU memory boundaries during inference operations. The MemoryAllocTest program is a standalone test harness that exercises the Triton C API's memory allocation pathways by running inference with configurable input/output memory placement (CPU vs. specific GPU devices), optional host policy routing, and comprehensive result validation including data type handling for INT32, FP32, and BYTES tensors.

Theoretical Basis

Why Memory Allocation Correctness Is Critical

Inference serving on heterogeneous hardware involves constant data movement between CPU system memory, pinned (page-locked) host memory, and GPU device memory across potentially multiple GPUs. A single bug in memory placement -- allocating on the wrong device, failing to copy data between address spaces, or misinterpreting memory type identifiers -- can cause silent data corruption, incorrect inference results, or outright crashes via CUDA illegal memory access errors. Memory allocation testing provides a systematic way to verify these pathways before production deployment.

Test Architecture

The test program operates as an in-process Triton server consumer (no network layer), directly invoking the TRITONSERVER C API:

Server creation: A TRITONSERVER_Server is instantiated with a specified model repository, strict model configuration, and explicit model control mode (loading only the model under test).
Memory specification: The user specifies input device (-i) and output device (-o) using integer device IDs where -1 indicates CPU.
Input preparation: Test data is generated in CPU memory and optionally copied to GPU memory via cudaMalloc and cudaMemcpy.
Inference execution: The request is submitted asynchronously via TRITONSERVER_ServerInferAsync with a promise/future pattern for synchronization.
Output validation: Results are retrieved, their memory type and device ID are verified against expectations, and arithmetic correctness is checked (add/subtract operations on known inputs).

Host Policy Testing

A particularly important test scenario involves NUMA-aware host policies. When a host policy name is specified (-h flag), the test submits two sets of input data:

Default input data containing zeros (the "wrong" data)
Host-policy-specific input data containing the actual test values

The test then verifies that the inference engine selected the correct input buffer based on the host policy, proving that the host policy routing mechanism works correctly. Conversely, when no host policy is specified, the test attaches zero data under a fake host policy name to verify that the default (non-policy) path is taken.

Data Type Coverage

The test exercises three fundamentally different data type pathways:

Data Type	Memory Layout	Validation Method
INT32	Fixed 4-byte elements, 16-element vectors	Arithmetic comparison (input0 +/- input1 = output0/output1)
FP32	Fixed 4-byte floating point elements	Floating point arithmetic comparison
BYTES	Variable-length strings with 4-byte length prefix	String-to-integer conversion and arithmetic comparison

BYTES (string) type testing is especially important because it exercises the variable-length serialization pathway where each element is prefixed with a uint32_t length field, a fundamentally different memory access pattern than fixed-size numeric types.

GPU Memory Lifecycle Management

GPU memory is managed through RAII-style std::unique_ptr with custom deleters that call cudaFree. The test carefully sets the active CUDA device before both allocation and deallocation, preventing cross-device memory management errors. This pattern mirrors what production inference code must do when managing buffers across multiple GPUs.

Response Allocator Verification

The test installs a custom ResponseAlloc callback that respects the IOSpec configuration to allocate output buffers in the requested memory type. This verifies that Triton's response allocator protocol correctly propagates memory type preferences from the client through to the buffer allocation layer, a critical contract for zero-copy inference pipelines.

Cross-Device Result Transfer

When outputs are placed in GPU memory, the test copies results back to CPU memory via cudaMemcpy(DeviceToHost) before performing validation. This exercises the full round-trip path that production clients must follow when consuming GPU-resident inference results.

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment