Principle:Kserve Kserve Prediction Protocol

Knowledge Sources	KServe V1 Protocol KServe V2 Protocol KFServing Predict Protocol
Domains	API_Design, Model_Serving, Inference
Last Updated	2026-02-13 00:00 GMT

Overview

A standardized request-response protocol for sending prediction inputs to ML model servers and receiving inference outputs via REST and gRPC.

Description

KServe defines two prediction protocols:

• V1 Protocol: TensorFlow Serving compatible. REST path: /v1/models/<model>:predict. Input format: {"instances": [...]}.
• V2 Protocol: Open Inference Protocol (formerly KFServing v2). REST path: /v2/models/<model>/infer. Supports tensor-based I/O with explicit shapes and data types.

Both protocols support REST (JSON) and gRPC (protobuf) transports. The protocol choice depends on the model framework and serving runtime configuration.

Usage

Use V1 protocol for TensorFlow Serving compatible models. Use V2 protocol for Triton, MLServer, and other frameworks supporting the Open Inference Protocol. HuggingFace runtime supports both.

Theoretical Basis

# V1 Protocol (TensorFlow Serving compatible)
POST /v1/models/<model_name>:predict
Request:  {"instances": [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]}
Response: {"predictions": [0, 1]}

# V2 Protocol (Open Inference Protocol)
POST /v2/models/<model_name>/infer
Request:  {"inputs": [{"name": "input-0", "shape": [2, 3], "datatype": "FP32",
           "data": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]}]}
Response: {"outputs": [{"name": "output-0", "shape": [2], "datatype": "INT64",
           "data": [0, 1]}]}

# gRPC Transport
Uses protobuf-defined PredictionService with Predict RPC
Supports streaming and binary tensor data

Related Pages

Implemented By

• Implementation:Kserve_Kserve_Prediction_Request_Protocol