Implementation:Kserve Kserve Graph Router Engine

Knowledge Sources	KServe GJSON
Domains	Pipeline, Routing, Request_Processing
Last Updated	2026-02-13 00:00 GMT

Overview

Concrete Go router binary that processes inference requests through graph nodes using routeStep, callService, pickupRoute, and pickupRouteByCondition functions.

Description

The graph router is a standalone Go binary (cmd/router/main.go) deployed as a pod by the InferenceGraph controller. It serves HTTP on port 8080 via graphHandler(). Incoming requests enter the root node and are dispatched by routeStep(), which handles each router type:

• Splitter: pickupRoute() uses crypto/rand for weighted random selection.
• Switch: pickupRouteByCondition() evaluates GJSON expressions via gjson.GetBytes().
• Ensemble: Parallel goroutines with result merging.
• Sequence: Serial execution with $request/$response data forwarding.

callService() makes HTTP POST requests to target services with header propagation.

Usage

The router binary is automatically deployed by the InferenceGraph controller. Users interact with it by sending HTTP POST requests to the graph's entry URL.

Code Reference

Source Location

• Repository: kserve
• File: cmd/router/main.go, Lines 243-547

Signature

// routeStep dispatches a request through a graph node
func routeStep(nodeName string, graph v1alpha1.InferenceGraphSpec,
    input []byte, headers http.Header) ([]byte, int, error)

// callService sends HTTP POST to a target service
func callService(serviceURL string, input []byte,
    headers http.Header) ([]byte, int, error)

// pickupRoute selects a step using weighted random (for Splitter)
func pickupRoute(routes []v1alpha1.InferenceStep) *v1alpha1.InferenceStep

// pickupRouteByCondition selects the first matching step (for Switch)
func pickupRouteByCondition(input []byte,
    routes []v1alpha1.InferenceStep) *v1alpha1.InferenceStep

// graphHandler serves as the HTTP handler for the router
func graphHandler(w http.ResponseWriter, req *http.Request)

Import

// This is a standalone binary, not a library
// Built via: go build -o router cmd/router/main.go

I/O Contract

Inputs

Name	Type	Required	Description
HTTP POST body	[]byte (JSON)	Yes	Inference input data
HTTP headers	http.Header	Yes	Propagated to downstream services (Istio mesh headers)
--graph-json	string (CLI arg)	Yes	Serialized InferenceGraphSpec

Outputs

Name	Type	Description
Sequence response	JSON	Final step output
Ensemble response	JSON	Merged: {"step1": resp1, "step2": resp2}
Splitter response	JSON	Selected step output wrapped: {"stepName": response}
Switch response	JSON	Matched step output

Usage Examples

Send Request to Graph

# Get graph URL
GRAPH_URL=$(kubectl get ig model-ensemble -o jsonpath='{.status.url}')

# Send inference request
curl -X POST ${GRAPH_URL} \
  -H "Content-Type: application/json" \
  -d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}'

# Ensemble response format:
# {"sklearn-iris": {"predictions": [1, 1]},
#  "xgboost-iris": {"predictions": [1, 1]}}

Related Pages

Implements Principle

• Principle:Kserve_Kserve_Graph_Request_Routing