Principle:Triton inference server Server Inference Request
| Knowledge Sources | |
|---|---|
| Domains | MLOps, Model_Serving, Inference |
| Last Updated | 2026-02-13 17:00 GMT |
Overview
The process of sending input tensor data to a deployed model and receiving output tensor predictions via a standardized inference protocol.
Description
Inference Request defines the client-server interaction pattern for model inference following the KServe v2 protocol. Clients construct a request specifying the target model, input tensors (with names, shapes, and data types), and desired outputs, then send it via HTTP POST or gRPC. The server validates the request, routes it to the correct model version, executes inference, and returns output tensors.
This principle supports both synchronous (single response) and asynchronous (streaming) inference patterns, as well as binary tensor data for efficient large payload transfer.
Usage
Use this principle whenever sending predictions to a deployed model on an inference server. It applies to all model types (classification, detection, generation, etc.) and all client transports (HTTP, gRPC). This is the core interaction pattern after a model has been deployed and verified as ready.
Theoretical Basis
The KServe v2 inference protocol:
POST /v2/models/<model>/versions/<version>/infer
Request JSON:
{
"id": "<request-id>",
"inputs": [
{ "name": "<name>", "shape": [<dims>], "datatype": "<type>", "data": [...] }
],
"outputs": [
{ "name": "<name>", "parameters": { "classification": <count> } }
]
}
Response JSON:
{
"id": "<request-id>",
"model_name": "<name>",
"model_version": "<version>",
"outputs": [
{ "name": "<name>", "shape": [<dims>], "datatype": "<type>", "data": [...] }
]
}