Principle:Triton inference server Server Model Namespacing Testing

Overview

Model Namespacing Testing validates the multi-tenant model repository isolation mechanism in Triton Inference Server. When --model-namespacing=true is enabled, models from different repositories are placed into separate namespaces, allowing identically named models to coexist without collision. This principle governs the test coverage required to ensure that namespace resolution, model duplication handling, dynamic model management, and ensemble cross-namespace references all function correctly under both POLL and EXPLICIT model control modes.

Theoretical Basis

Model namespacing addresses a fundamental problem in multi-tenant inference serving: name collision. In production deployments, organizations frequently need to serve multiple versions or variants of models that share the same name but originate from different repositories, teams, or deployment pipelines. Without namespacing, loading two repositories that each contain a model named "resnet50" would result in a conflict, forcing operators to rename models or maintain a single monolithic repository.

The namespace abstraction draws from well-established software engineering concepts: package namespaces in programming languages, schema isolation in databases, and Kubernetes namespaces in container orchestration. The key invariant is that a model's fully-qualified identity is (namespace, model_name) rather than just model_name, where the namespace is derived from the model's source repository.

Duplication detection: When namespacing is enabled, models with the same name in different repositories are allowed to coexist. The test_duplication suite validates that both models load successfully, each maintaining its own independent weights, configuration, and version history. Without namespacing, the server must detect and report the name collision.

Dynamic resolution: The test_dynamic_resolution suite tests runtime model management operations (load, unload, reload) in the presence of namespaced models. Under POLL mode, the server watches repository directories for changes; under EXPLICIT mode, the control API is used. Both modes must correctly resolve namespace-qualified model references and must not confuse models across namespaces during lifecycle transitions.

Ensemble cross-namespace references: Ensembles can reference composing models that reside in different namespaces. The test_ensemble_duplication suite validates that ensemble pipeline steps correctly resolve their model references within the namespace hierarchy, even when composing models have the same name in different repositories.

Non-duplication baseline: The test_no_duplication suite validates the baseline case where repositories contain distinct model names, confirming that namespacing does not introduce overhead or behavioral changes when no collisions exist.

POLL vs. EXPLICIT mode interactions: Under POLL mode, the server periodically scans repositories for changes. A critical concern is that Python bytecode caching (__pycache__ directories) created during model loading could be detected as repository changes, triggering unintended model reloads. The test infrastructure sets PYTHONDONTWRITEBYTECODE=1 to prevent this, and the tests validate that POLL mode correctly detects intentional changes while ignoring spurious filesystem activity. Under EXPLICIT mode, all model lifecycle operations go through the control API, and the tests validate that namespace-qualified load and unload requests are dispatched correctly.

Implementation Details

The test infrastructure creates parallel directory structures simulating multiple model repositories, each containing Python-based models (addsub and subadd arithmetic models). The test shell script iterates over all test scenarios under both POLL and EXPLICIT model control modes, starting a fresh server instance for each combination. The Python test driver (test.py) issues inference requests and model management API calls, verifying that responses come from the correct namespace-specific model instance and that lifecycle operations affect only the targeted namespace.

The repository structure is copied to a temporary test_dir for each test run, allowing the test to modify the directory structure (simulating model additions and removals) without affecting other test scenarios.

Related Pages

Implementation:Triton_inference_server_Server_L0_Model_Namespacing_Test Triton_inference_server_Server