Implementation:Triton inference server Server SagemakerServer

Knowledge Sources	Triton Inference Server Triton SageMaker
Domains	Cloud_Integration, AWS
Last Updated	2026-02-13 17:00 GMT

Overview

Concrete tool for serving inference through AWS SageMaker-compatible HTTP endpoints, supporting both single-model and multi-model endpoint (MME) modes.

Description

The SagemakerAPIServer class extends HTTPAPIServer to provide SageMaker-compatible routes (/ping, /invocations, /models). It supports environment-variable-driven configuration (SAGEMAKER_TRITON_DEFAULT_MODEL_NAME, SAGEMAKER_MULTI_MODEL), regex-based URL routing, and a mutex-protected model registry. The nested SagemakeInferRequestClass handles SageMaker-specific response formatting with appropriate headers.

Usage

Activated when Triton is launched in SageMaker mode (via --allow-sagemaker=true). Used when deploying Triton as a SageMaker endpoint, either for single model hosting or multi-model endpoints with dynamic model loading.

Code Reference

Source Location

• Repository: Triton Inference Server
• File: src/sagemaker_server.h
• Lines: 1-181
• File: src/sagemaker_server.cc
• Lines: 1-1064

Signature

class SagemakerAPIServer : public HTTPAPIServer {
 public:
  static TRITONSERVER_Error* Create(
      const std::shared_ptr<TRITONSERVER_Server>& server,
      triton::server::TraceManager* trace_manager,
      const std::shared_ptr<SharedMemoryManager>& smm,
      const TritonServerParameters& params,
      int32_t port, int thread_cnt,
      std::unique_ptr<HTTPServer>* http_server);

  class SagemakeInferRequestClass : public InferRequestClass {
   public:
    explicit SagemakeInferRequestClass(
        TRITONSERVER_Server* server, evhtp_request_t* req,
        DataCompressor::Type type);
  };

 private:
  void SagemakerMMELoadModel(evhtp_request_t* req);
  void SagemakerMMEUnloadModel(evhtp_request_t* req);
  void SagemakerMMEListModel(evhtp_request_t* req);
  void SagemakerMMEGetModel(evhtp_request_t* req);

  std::mutex mu_;
  std::unordered_map<std::string, std::string> sagemaker_models_list_;
};

Import

#include "sagemaker_server.h"

I/O Contract

Inputs

Name	Type	Required	Description
SAGEMAKER_TRITON_DEFAULT_MODEL_NAME	env var	Yes (single)	Default model for /invocations
SAGEMAKER_MULTI_MODEL	env var	No	Set "true" for MME mode
X-Amzn-SageMaker-Target-Model	HTTP header	Yes (MME)	Target model in multi-model mode

Outputs

Name	Type	Description
/ping	HTTP 200	Health check response
/invocations	HTTP JSON	Inference results in SageMaker format
/models	HTTP JSON	Model listing (MME mode)

Usage Examples

SageMaker Single Model

# Start Triton in SageMaker mode
export SAGEMAKER_TRITON_DEFAULT_MODEL_NAME=my_model
tritonserver --model-repository=/models --allow-sagemaker=true --sagemaker-port=8080

# Send inference request
curl -X POST http://localhost:8080/invocations \
  -H "Content-Type: application/json" \
  -d '{"inputs": [{"name": "input0", "data": [1, 2, 3]}]}'

SageMaker Multi-Model Endpoint

# Start in MME mode
export SAGEMAKER_MULTI_MODEL=true
tritonserver --model-repository=/models --allow-sagemaker=true

# Load a model
curl -X POST http://localhost:8080/models \
  -H "Content-Type: application/json" \
  -d '{"model_name": "model_a", "url": "/models/model_a"}'

# Invoke specific model
curl -X POST http://localhost:8080/invocations \
  -H "X-Amzn-SageMaker-Target-Model: model_a" \
  -d '{"inputs": [...]}'

Related Pages

• Environment:Triton_inference_server_Server_GPU_CUDA_Runtime