Implementation:Triton inference server Server VertexAiServer

Knowledge Sources	Triton Inference Server
Domains	Cloud_Integration, GCP
Last Updated	2026-02-13 17:00 GMT

Overview

Concrete tool for serving inference through Google Cloud Vertex AI-compatible HTTP endpoints, translating Vertex AI API requests to Triton's internal endpoints.

Description

The VertexAiAPIServer class extends HTTPAPIServer to provide Vertex AI-compatible prediction and health routes. It implements a redirect mechanism that maps POST requests to appropriate Triton API endpoints (inference, metadata, health, shared memory, model repository, tracing, logging, statistics) using the X-Vertex-Ai-Triton-Redirect header. It supports header-length-based JSON/binary request separation and a configurable default model name.

Usage

Activated when Triton is launched with --allow-vertex-ai=true. Used when deploying Triton as a Google Cloud Vertex AI prediction endpoint.

Code Reference

Source Location

• Repository: Triton Inference Server
• File: src/vertex_ai_server.h
• Lines: 1-82
• File: src/vertex_ai_server.cc
• Lines: 1-355

Signature

class VertexAiAPIServer : public HTTPAPIServer {
 public:
  static TRITONSERVER_Error* Create(
      const std::shared_ptr<TRITONSERVER_Server>& server,
      triton::server::TraceManager* trace_manager,
      const std::shared_ptr<SharedMemoryManager>& smm,
      const TritonServerParameters& params,
      int32_t port, int thread_cnt,
      std::unique_ptr<HTTPServer>* http_server);

 private:
  void Handle(evhtp_request_t* req) override;
  std::string default_model_name_;
};

Import

#include "vertex_ai_server.h"

I/O Contract

Inputs

Name	Type	Required	Description
AIP_MODE	env var	Yes	Set to "VERTEX_AI_PREDICTION" by Vertex AI platform
X-Vertex-Ai-Triton-Redirect	HTTP header	No	Redirect path to specific Triton API
default_model_name	config	No	Default model for predictions

Outputs

Name	Type	Description
/v1/models/{model}:predict	HTTP JSON	Prediction results
/v1/models/{model}	HTTP JSON	Model metadata
/health	HTTP 200	Health check

Usage Examples

Deploy on Vertex AI

# Start Triton in Vertex AI mode
tritonserver --model-repository=/models \
  --allow-vertex-ai=true \
  --vertex-ai-port=8080 \
  --vertex-ai-default-model=my_model

# Prediction request
curl -X POST http://localhost:8080/v1/models/my_model:predict \
  -H "Content-Type: application/json" \
  -d '{"instances": [{"input": [1, 2, 3]}]}'

# Access Triton-specific API via redirect
curl -X POST http://localhost:8080/v1/models/my_model:predict \
  -H "X-Vertex-Ai-Triton-Redirect: /v2/models/my_model/stats"

Related Pages

• Environment:Triton_inference_server_Server_GPU_CUDA_Runtime