Principle:Kserve Kserve Graph Request Routing

Knowledge Sources	KServe InferenceGraph GJSON Syntax
Domains	Pipeline, Routing, Request_Processing
Last Updated	2026-02-13 00:00 GMT

Overview

A runtime request processing engine that routes inference requests through graph nodes by dispatching to steps based on the node's router type.

Description

Graph Request Routing is the runtime execution of an InferenceGraph. When a request arrives at the graph router pod, it enters the root node and is dispatched according to the node's router type:

• Sequence: Executes steps serially, chaining outputs to inputs.
• Ensemble: Launches all steps as concurrent goroutines, merges results.
• Splitter: Uses cryptographic random weighted selection to pick one step.
• Switch: Evaluates GJSON conditions sequentially, routes to first match.

Steps targeting other graph nodes trigger recursive routing. Steps targeting services make HTTP POST calls with header propagation (including Istio mesh headers).

Usage

This principle is active whenever an InferenceGraph receives a request. Understanding the routing engine is essential for debugging pipeline behavior, optimizing latency, and designing graph topologies.

Theoretical Basis

# Routing engine algorithm (NOT implementation code)
function routeStep(nodeName, graph, input, headers):
    node = graph.nodes[nodeName]

    switch node.routerType:
        case Sequence:
            response = input
            for step in node.steps:
                stepInput = step.data == "$request" ? input : response
                response = executeStep(step, stepInput, headers)
            return response

        case Ensemble:
            results = parallel_map(node.steps, step =>
                executeStep(step, input, headers))
            return merge(results)  # {"stepName": response, ...}

        case Splitter:
            step = weightedRandomSelect(node.steps)
            return executeStep(step, input, headers)

        case Switch:
            for step in node.steps:
                if gjson.match(input, step.condition):
                    return executeStep(step, input, headers)
            return 404

function executeStep(step, input, headers):
    if step.nodeName:
        return routeStep(step.nodeName, graph, input, headers)
    else:
        return httpPost(step.serviceURL, input, headers)

Related Pages

Implemented By

• Implementation:Kserve_Kserve_Graph_Router_Engine