Principle:SeldonIO Seldon core Multi Modal Pipeline Composition

Field	Value
Overview	Composing HuggingFace models with different modalities (speech, text) into a unified inference pipeline.
Domains	NLP, Data_Flow
Related Implementation	SeldonIO_Seldon_core_Seldon_Pipeline_CRD_Multi_Modal
Knowledge Sources	Repo (https://github.com/SeldonIO/seldon-core), Doc (https://docs.seldon.io/projects/seldon-core/en/v2/)
Last Updated	2026-02-13 00:00 GMT

Description

Multi-modal pipelines chain models that process different data types. For example, a speech-to-sentiment pipeline chains Whisper (audio to text) with a sentiment model (text to labels). The tensorMap field remaps output tensor names between models with different naming conventions (e.g., whisper's "output" becomes sentiment's "args").

A multi-modal pipeline in Seldon Core 2 is defined by:

1. Steps -- each step references a deployed Model and optionally declares input dependencies and tensor name remappings
2. Input dependencies -- the inputs field on each step specifies which upstream step's output feeds into it
3. Tensor remapping -- the tensorMap field translates tensor names between models with incompatible interfaces
4. Multi-output -- the output.steps field can expose results from multiple pipeline stages (e.g., both the transcription and sentiment result)

The pipeline DAG can also include auxiliary models such as:

• Input transform models that preprocess data between stages
• Explainer models that provide interpretability for classification results

Theoretical Basis

Multi-modal pipelines decompose complex tasks into modality-specific stages. Each stage is optimized for its modality (speech recognition, NLP classification). The pipeline DAG defines data dependencies, and tensor remapping handles interface mismatches between independently developed models.

The key design principles are:

• Modular composition -- each model handles exactly one modality or task, enabling independent development, testing, and scaling
• Interface adaptation -- the tensorMap mechanism bridges naming conventions between models that were not originally designed to work together
• Asynchronous data flow -- Seldon Core 2 uses Kafka for inter-step communication, enabling each step to process at its own rate without blocking others
• Multi-output aggregation -- by exposing multiple pipeline steps in the output, clients receive results from all stages without needing separate requests

The tensor remapping follows the format {step}.outputs.{tensor_name}: {target_name}, where:

• {step} is the name of the upstream step
• {tensor_name} is the output tensor name from the upstream model
• {target_name} is the input tensor name expected by the downstream model

Usage

This principle applies when building inference pipelines that combine multiple HuggingFace models across different modalities, including:

• Chaining speech-to-text with text classification (e.g., speech-to-sentiment)
• Combining text generation with post-processing models
• Adding explainability models as pipeline branches alongside base classifiers
• Any scenario where independently developed models need to be composed into a unified workflow

Related Pages

• SeldonIO_Seldon_core_Seldon_Pipeline_CRD_Multi_Modal -- implements this principle with a concrete speech-to-sentiment pipeline YAML
• SeldonIO_Seldon_core_HuggingFace_Text_Inference -- builds on single-model inference to compose multi-step inference chains
• SeldonIO_Seldon_core_Model_Explainability_Integration -- extends pipeline composition with explainability branches
• SeldonIO_Seldon_core_Pipeline_Topology_Definition -- generalizes pipeline DAG definition for all model types

Implementation:SeldonIO_Seldon_core_Seldon_Pipeline_CRD_Multi_Modal