Principle:Kserve Kserve Graph Topology Design

Knowledge Sources	KServe InferenceGraph DAG Pipeline Patterns
Domains	Pipeline, Architecture, Graph_Theory
Last Updated	2026-02-13 00:00 GMT

Overview

A design pattern for composing multi-model inference workflows using four fundamental routing types: Sequence, Ensemble, Splitter, and Switch.

Description

Graph Topology Design is the process of selecting the appropriate routing pattern for a multi-model inference pipeline. KServe provides four node types:

• Sequence: Chains steps serially, passing output of one to input of the next.
• Ensemble: Fans out to all steps in parallel, merges responses into a keyed JSON object.
• Splitter: Routes each request to exactly one step based on weighted random selection (weights must sum to 100).
• Switch: Evaluates GJSON conditions in order, routes to the first matching step (returns 404 if no match).

Each node type can be combined in a DAG (Directed Acyclic Graph) with a mandatory root node.

Usage

Choose the topology based on pipeline requirements:

• Sequence for serial pre-processing → prediction → post-processing chains
• Ensemble for aggregating predictions from multiple models
• Splitter for A/B testing or load balancing between model versions
• Switch for conditional routing based on input features

Theoretical Basis

# Graph topology patterns (NOT implementation code)

Sequence: A → B → C
  - Output of A becomes input of B
  - $request forwards original input, $response forwards step output

Ensemble: A ──→ B
              ├→ C    → merge({B: respB, C: respC})
  - All steps execute in parallel
  - Responses merged into single JSON object

Splitter: A ──→ B (70%)
              ├→ C (30%)    weights sum to 100
  - Cryptographic random selection per request
  - Exactly one step receives each request

Switch: A ──→ B if condition_1 matches
            ├→ C if condition_2 matches
  - GJSON conditions evaluated in order
  - First match wins
  - 404 if no conditions match

Related Pages

Implemented By

• Implementation:Kserve_Kserve_InferenceRouterType_Enum