Environment:SeldonIO Seldon core Kafka Messaging Environment

Knowledge Sources	Seldon Core 2 Kafka Configuration Managed Kafka
Domains	Infrastructure, Messaging
Last Updated	2026-02-13 14:00 GMT

Overview

Apache Kafka 3.8.1 messaging cluster with configurable topics, consumer groups, and security (PLAINTEXT/SSL/SASL) for Seldon Core 2 data plane communication.

Description

Seldon Core 2 uses Apache Kafka as its core data plane. All inference requests for pipelines flow through Kafka topics, enabling asynchronous, scalable model chaining. Each model and pipeline gets dedicated input/output topics following the naming pattern `{prefix}.{namespace}.model.{name}.inputs`. The Kafka cluster can be self-hosted (Bitnami, Strimzi) or managed (AWS MSK, Confluent Cloud, Azure Event Hub). Security supports PLAINTEXT, SSL/TLS, SASL_SSL (SCRAM, PLAIN, OAuth).

Usage

Use this environment for any Seldon Core 2 deployment that uses pipelines. Kafka is a mandatory dependency for the data plane. Even single-model deployments require Kafka for the model gateway to manage requests. The Kafka cluster must be running and accessible before deploying Seldon Core 2 components.

System Requirements

Category	Requirement	Notes
Kafka	Apache Kafka 3.8.1	Bitnami image used for local; managed Kafka for production
Zookeeper	3.8	Required for Bitnami Kafka; managed Kafka handles internally
Memory	512Mi - 1Gi per broker	Depends on topic count and retention
Disk	Variable	Depends on message retention policy and throughput
Network	Low latency to Seldon components	Higher latency = lower pipeline throughput

Dependencies

Self-Hosted (Local/Dev)

`docker.io/bitnami/kafka:3.8.1`
`docker.io/bitnami/zookeeper:3.8`

Self-Hosted (Kubernetes)

Strimzi Kafka Operator (recommended for K8s self-hosted)
Helm chart for Strimzi deployment

Managed Kafka Providers

AWS MSK (mTLS or SASL authentication)
Confluent Cloud (SASL/PLAIN or OAuth 2.0)
Azure Event Hub (SASL/PLAIN via Kafka protocol)

Credentials

For PLAINTEXT (local dev): No credentials required.

For SSL/TLS (encrypted):

`KAFKA_CLIENT_TLS_SECRET_NAME`: K8s secret with client certificates
`KAFKA_BROKER_TLS_SECRET_NAME`: K8s secret with broker CA certificate

For SASL_SSL (authenticated):

`KAFKA_CLIENT_SASL_USERNAME`: SASL username (default: "seldon")
`KAFKA_CLIENT_SASL_SECRET_NAME`: K8s secret containing SASL password
`KAFKA_CLIENT_SASL_PASSWORD_LOCATION`: Path to password file within secret

For Confluent Schema Registry (optional):

`SCHEMA_REGISTRY_CONFIG_PATH`: Path to schema registry configuration file

Quick Install

# Self-hosted on Kubernetes with Strimzi
kubectl create namespace kafka
kubectl apply -f https://strimzi.io/install/latest?namespace=kafka

# Create a Kafka cluster
kubectl apply -f - <<EOF
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: seldon-kafka
  namespace: kafka
spec:
  kafka:
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
EOF

Code Evidence

Kafka configuration defaults from `k8s/helm-charts/seldon-core-v2-setup/values.yaml`:

kafka:
  bootstrap: seldon-kafka-bootstrap.seldon-mesh:9092
  topicPrefix: seldon
  consumer:
    sessionTimeoutMs: 6000
    autoOffsetReset: earliest
    topicMetadataRefreshIntervalMs: 1000
    topicMetadataPropagationMaxMs: 300000
    messageMaxBytes: 1000000000
  producer:
    lingerMs: 0
    messageMaxBytes: 1000000000
  topics:
    replicationFactor: 1
    numPartitions: 1

Kafka security protocol handling from `components/kafka/config.go`:

// Supports: PLAINTEXT, SSL, SASL_SSL
switch kafkaSecurityProtocol {
case "SSL":
    // Configure TLS certificates
case "SASL_SSL":
    // Configure SASL + TLS
default:
    // PLAINTEXT - no security
}

Topic naming pattern from `docs-gb/managing-kafka-topics.md`:

{prefix}.{namespace}.model.{name}.inputs
{prefix}.{namespace}.model.{name}.outputs
{prefix}.{namespace}.pipeline.{name}.inputs
{prefix}.{namespace}.pipeline.{name}.outputs
{prefix}.{namespace}.errors.errors

Common Errors

Error Message	Cause	Solution
`Broker not available`	Kafka bootstrap address incorrect or Kafka not running	Verify Kafka pods are running; check `kafka.bootstrap` in Helm values
`Topic metadata refresh timeout`	Kafka slow to propagate topic metadata	Increase `topicMetadataPropagationMaxMs` (default 300000ms = 5min)
`SASL authentication failed`	Wrong credentials or mechanism	Verify SASL username/password; check mechanism matches broker config
`SSL handshake failed`	Certificate mismatch or expired	Check TLS certificates in K8s secrets; verify CA chain
`Message too large`	Payload exceeds `messageMaxBytes`	Increase `messageMaxBytes` in both consumer and producer config (default 1GB)

Compatibility Notes

Topic isolation: Use `topicPrefix` and `consumerGroupIdPrefix` to isolate Seldon topics when sharing a Kafka cluster with other applications.
Topic cleanup: Set `cleanTopicsOnDelete: true` in pipeline/model CRDs to auto-delete topics on unload. Default is `false` (topics retained).
Managed Kafka: AWS MSK requires IAM or mTLS; Confluent Cloud uses SASL/PLAIN or OAuth; Azure Event Hub uses SASL/PLAIN with connection string.
Schema Registry: Optional Confluent Schema Registry integration available for centralized schema management.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment