Jump to content

Connect Leeroopedia MCP: Equip your AI agents to search best practices, build plans, verify code, diagnose failures, and look up hyperparameter defaults.

Workflow:Apache Shardingsphere Shadow Rule Configuration

From Leeroopedia


Metadata

Field Value
Workflow Name Shadow Rule Configuration
Project Apache ShardingSphere
Component Shadow Database Feature
Domains Distributed_Database, Shadow_Testing, Configuration
Trigger YAML configuration file loaded at application startup or dynamic DistSQL change event
Entry Point YamlShadowRuleConfiguration (YAML deserialization)
Exit Point ShadowRule (fully initialized rule object)
SPI Order 55 (ShadowOrder.ORDER)
Knowledge Sources Repo, Doc
Last Updated 2026-02-10 12:00 GMT

Overview

The Shadow Rule Configuration workflow defines the end-to-end process by which Apache ShardingSphere transforms a user-authored YAML shadow database configuration into a fully operational ShadowRule runtime object. Shadow database functionality enables routing SQL traffic to a shadow (test) database instead of the production database based on hint-based or column-based algorithms. This is commonly used for full-link stress testing in production environments without affecting real data.

The workflow spans three architectural layers of the shadow feature module:

  • API layer (features/shadow/api): Defines the Java configuration model -- ShadowRuleConfiguration, ShadowDataSourceConfiguration, and ShadowTableConfiguration.
  • Core layer (features/shadow/core): Implements YAML representations, bidirectional swappers, validation checkers, the SPI rule builder, and the runtime rule objects.
  • Dynamic change layer (features/shadow/core): Provides changed-item processors for hot-reloading individual configuration elements without full restart.

The configuration pipeline follows a six-stage process: YAML definition, YAML deserialization, object conversion (swapping), validation, rule construction, and runtime initialization.

Description

Configuration Model

The shadow rule configuration consists of three primary sections:

Data Sources map logical names to production/shadow data source pairs. Each ShadowDataSourceConfiguration holds:

  • name -- Logical identifier for the data source mapping
  • productionDataSourceName -- Reference to the production data source
  • shadowDataSourceName -- Reference to the shadow data source

Tables define per-table shadow routing rules. Each ShadowTableConfiguration holds:

  • dataSourceNames -- Collection of logical data source names applicable to this table
  • shadowAlgorithmNames -- Collection of algorithm names governing shadow determination for this table

Shadow Algorithms are named AlgorithmConfiguration entries specifying the algorithm type and properties. Two categories exist:

  • HintShadowAlgorithm -- Routes based on SQL hints (e.g., SQL_HINT type)
  • ColumnShadowAlgorithm -- Routes based on shadow column values, categorized by ShadowOperationType (INSERT, UPDATE, DELETE, SELECT)

An optional defaultShadowAlgorithmName designates a fallback algorithm. When specified, it must reference an algorithm of type SQL_HINT.

YAML Representation

The YAML layer mirrors the Java configuration model with annotation-driven metadata:

YAML Class Annotation Purpose
YamlShadowRuleConfiguration @RuleNodeTupleEntity("shadow") Top-level rule configuration for YAML
YamlShadowDataSourceConfiguration (via @RuleNodeTupleField(type = Type.DATA_SOURCE) on parent) YAML data source mapping
YamlShadowTableConfiguration (via @RuleNodeTupleField(type = Type.TABLE) on parent) YAML table configuration
YamlAlgorithmConfiguration (via @RuleNodeTupleField(type = Type.ALGORITHM) on parent) YAML algorithm configuration

The defaultShadowAlgorithmName field is annotated with @RuleNodeTupleField(type = Type.DEFAULT_ALGORITHM).

Swapper Architecture

The YamlShadowRuleConfigurationSwapper implements YamlRuleConfigurationSwapper and orchestrates bidirectional conversion using three delegate swappers:

Swapper Direction Behavior
YamlShadowDataSourceConfigurationSwapper Java to YAML Maps productionDataSourceName and shadowDataSourceName fields
YamlShadowDataSourceConfigurationSwapper YAML to Java Throws UnsupportedOperationException (conversion handled inline by parent swapper)
YamlShadowTableConfigurationSwapper Bidirectional Converts dataSourceNames and shadowAlgorithmNames collections
YamlAlgorithmConfigurationSwapper Bidirectional Converts algorithm type and properties

During the swapToObject conversion, the parent swapper applies two defaulting rules:

  1. Default data source assignment: If exactly one data source mapping exists and a table has no explicit data source names, the single data source name is auto-assigned.
  2. Default algorithm assignment: If a defaultShadowAlgorithmName is set and a table has no explicit algorithm names, the default is auto-assigned.

Validation Pipeline

ShadowRuleConfigurationChecker implements DatabaseRuleConfigurationChecker and performs a five-stage validation:

  1. Algorithm SPI validation -- Verifies each shadow algorithm type is loadable via TypedSPILoader.checkService(ShadowAlgorithm.class, ...).
  2. Default algorithm validation -- If defaultShadowAlgorithmName is non-null, confirms the referenced algorithm exists and is of type SQL_HINT. Throws NotImplementHintShadowAlgorithmException on failure.
  3. Data source existence validation -- Confirms both productionDataSourceName and shadowDataSourceName exist in the database's data source map. Throws MissingRequiredProductionDataSourceException or MissingRequiredShadowDataSourceException on failure.
  4. Table-to-data-source reference validation -- Ensures every data source name referenced by a table exists in the declared shadow data sources. Throws ShadowDataSourceMappingNotFoundException on failure.
  5. Table-to-algorithm reference validation -- Ensures every table has at least one shadow algorithm name, and every referenced algorithm name exists in the declared shadow algorithms. Throws MissingRequiredAlgorithmException on failure.

A separate ShadowRuleConfigurationEmptyChecker detects empty configurations by checking whether the data sources or tables collections are empty.

Rule Construction

ShadowRuleBuilder implements DatabaseRuleBuilder<ShadowRuleConfiguration> as an SPI extension and delegates directly to the ShadowRule constructor: 

public ShadowRule build(final ShadowRuleConfiguration ruleConfig, ...) {
    return new ShadowRule(ruleConfig);
}

Runtime Rule Objects

The ShadowRule constructor initializes four components:

  1. shadowAlgorithms -- Map<String, ShadowAlgorithm> created by loading each algorithm via TypedSPILoader.getService(ShadowAlgorithm.class, type, props).
  2. defaultShadowAlgorithm -- Resolved from the map using defaultShadowAlgorithmName, may be null.
  3. dataSourceRules -- CaseInsensitiveMap<String, ShadowDataSourceRule> mapping logical names to production/shadow data source pairs.
  4. tableRules -- CaseInsensitiveMap<String, ShadowTableRule> where each ShadowTableRule classifies its algorithms into:
    • hintShadowAlgorithmNames -- Algorithms implementing HintShadowAlgorithm
    • columnShadowAlgorithmNames -- EnumMap<ShadowOperationType, Collection<ShadowAlgorithmNameRule>> mapping operation types to column/algorithm pairs

A ShadowDataSourceMapperRuleAttribute is registered as a RuleAttributes entry to expose data source mappings to the ShardingSphere infrastructure.

Dynamic Change Processors

Four SPI processors handle runtime configuration changes without restart:

Processor Rule Changed Item Type Behavior
ShadowAlgorithmChangedProcessor ("shadow", "shadow_algorithms") Adds/removes algorithm entries in shadowAlgorithms map
DefaultShadowAlgorithmNameChangedProcessor ("shadow", "default_shadow_algorithm_name") Sets/clears the defaultShadowAlgorithmName field
ShadowDataSourceChangedProcessor ("shadow", "data_sources") Adds/removes data source mappings by name with replace-on-conflict semantics
ShadowTableChangedProcessor ("shadow", "tables") Puts/removes table configuration entries by table name

Usage

A typical YAML configuration for the shadow rule: 

rules:
  - !SHADOW
    dataSources:
      shadow-data-source:
        productionDataSourceName: ds_prod
        shadowDataSourceName: ds_shadow
    tables:
      t_order:
        dataSourceNames:
          - shadow-data-source
        shadowAlgorithmNames:
          - user-id-insert-match-algorithm
          - sql-hint-algorithm
    shadowAlgorithms:
      user-id-insert-match-algorithm:
        type: VALUE_MATCH
        props:
          operation: insert
          column: user_id
          value: 0
      sql-hint-algorithm:
        type: SQL_HINT
    defaultShadowAlgorithmName: sql-hint-algorithm

This configuration:

  • Declares one data source mapping (shadow-data-source) linking ds_prod to ds_shadow.
  • Configures the t_order table to use both a column-based match algorithm and a hint-based algorithm.
  • Sets sql-hint-algorithm as the default fallback algorithm for tables without explicit algorithm assignments.

Execution Steps

Step 1: Define Shadow Configuration in YAML

Aspect Detail
Action Author the YAML configuration file with the !SHADOW rule tag
Input User-defined YAML file specifying data sources, tables, shadow algorithms, and optional default algorithm name
Output Raw YAML text conforming to the YamlShadowRuleConfiguration schema
Key Fields dataSources (map of name to production/shadow pair), tables (map of table name to data source and algorithm references), shadowAlgorithms (map of algorithm name to type and properties), defaultShadowAlgorithmName (optional string)

Step 2: YAML Deserialized to YamlShadowRuleConfiguration

Aspect Detail
Action ShardingSphere's YAML engine parses the !SHADOW tagged block into typed Java objects
Input Raw YAML text
Output YamlShadowRuleConfiguration instance containing Map<String, YamlShadowDataSourceConfiguration>, Map<String, YamlShadowTableConfiguration>, Map<String, YamlAlgorithmConfiguration>, and defaultShadowAlgorithmName
Classes Involved YamlShadowRuleConfiguration, YamlShadowDataSourceConfiguration, YamlShadowTableConfiguration, YamlAlgorithmConfiguration
Annotations @RuleNodeTupleEntity("shadow") on the root class; @RuleNodeTupleField with types DATA_SOURCE, TABLE, ALGORITHM, DEFAULT_ALGORITHM on fields

Step 3: Swapper Converts YAML Objects to Java Configuration

Aspect Detail
Action YamlShadowRuleConfigurationSwapper.swapToObject() converts the YAML model into the API configuration model
Input YamlShadowRuleConfiguration
Output ShadowRuleConfiguration containing Collection<ShadowDataSourceConfiguration>, Map<String, ShadowTableConfiguration>, Map<String, AlgorithmConfiguration>
Key Logic (1) Each YamlShadowDataSourceConfiguration is converted to ShadowDataSourceConfiguration with the map key as the name. (2) Table and algorithm entries are converted via delegate swappers. (3) Default data source auto-assignment: if exactly one data source exists, tables with empty dataSourceNames receive it. (4) Default algorithm auto-assignment: tables with empty shadowAlgorithmNames receive defaultShadowAlgorithmName.
Delegate Swappers YamlShadowDataSourceConfigurationSwapper, YamlShadowTableConfigurationSwapper, YamlAlgorithmConfigurationSwapper

Step 4: Configuration Checker Validates Completeness and Correctness

Aspect Detail
Action ShadowRuleConfigurationChecker.check() executes a five-stage validation pipeline
Input ShadowRuleConfiguration, database name, data source map, built rules
Output Validation passes silently or throws a specific exception
Stage 1 Algorithm SPI check: Calls TypedSPILoader.checkService(ShadowAlgorithm.class, type, props) for each algorithm
Stage 2 Default algorithm check: If defaultShadowAlgorithmName is non-null, confirms it exists and has type SQL_HINT
Stage 3 Data source existence check: Verifies both production and shadow data source names exist in the provided data source map
Stage 4 Table-data source reference check: Confirms each table's referenced data source names exist in the declared data source configurations
Stage 5 Table-algorithm reference check: Confirms each table has at least one algorithm name and all referenced names exist in declared algorithms
Pre-check ShadowRuleConfigurationEmptyChecker.isEmpty() returns true if dataSources or tables is empty, short-circuiting rule creation

Step 5: ShadowRuleBuilder Constructs ShadowRule from Configuration

Aspect Detail
Action ShadowRuleBuilder.build() invokes new ShadowRule(ruleConfig)
Input Validated ShadowRuleConfiguration, database name, protocol type, resource metadata, built rules, compute node instance context
Output ShadowRule instance
SPI Interface DatabaseRuleBuilder<ShadowRuleConfiguration>
SPI Order 55 (ShadowOrder.ORDER)
Note The builder delegates entirely to the ShadowRule constructor; context parameters (database name, protocol type, etc.) are not used during construction

Step 6: ShadowRule Initializes Algorithms, Table Rules, and Data Source Mappings

Aspect Detail
Action The ShadowRule constructor performs four initialization steps
Input ShadowRuleConfiguration
Output Fully populated ShadowRule with algorithms, data source rules, table rules, and rule attributes
Init 1: shadowAlgorithms Iterates shadowAlgorithms config map, loads each via TypedSPILoader.getService(ShadowAlgorithm.class, type, props), stores in LinkedHashMap
Init 2: defaultShadowAlgorithm Looks up defaultShadowAlgorithmName in the loaded algorithms map; may be null
Init 3: dataSourceRules Converts each ShadowDataSourceConfiguration to ShadowDataSourceRule(productionDataSource, shadowDataSource), stored in a CaseInsensitiveMap
Init 4: tableRules Creates ShadowTableRule for each table, which classifies algorithms into hintShadowAlgorithmNames (for HintShadowAlgorithm instances) and columnShadowAlgorithmNames (an EnumMap<ShadowOperationType, Collection<ShadowAlgorithmNameRule>> for ColumnShadowAlgorithm instances)
Attributes Registers ShadowDataSourceMapperRuleAttribute exposing production-to-shadow data source mappings to the ShardingSphere infrastructure

Execution Diagram

                    +---------------------------+
                    |   YAML Configuration File |
                    |       (!SHADOW tag)        |
                    +-------------+-------------+
                                  |
                          Step 1: Define
                                  |
                                  v
                    +---------------------------+
                    |    YAML Engine Parsing     |
                    +-------------+-------------+
                                  |
                       Step 2: Deserialize
                                  |
                                  v
              +-------------------------------------------+
              |     YamlShadowRuleConfiguration           |
              |  +-------------------------------------+  |
              |  | dataSources: Map<String,             |  |
              |  |   YamlShadowDataSourceConfiguration>|  |
              |  | tables: Map<String,                  |  |
              |  |   YamlShadowTableConfiguration>     |  |
              |  | shadowAlgorithms: Map<String,        |  |
              |  |   YamlAlgorithmConfiguration>        |  |
              |  | defaultShadowAlgorithmName: String   |  |
              |  +-------------------------------------+  |
              +---------------------+---------------------+
                                    |
                         Step 3: Swap (Convert)
                                    |
                                    v
              +-------------------------------------------+
              |  YamlShadowRuleConfigurationSwapper       |
              |  .swapToObject()                          |
              |                                           |
              |  Delegates:                               |
              |  - YamlShadowDataSourceConfigSwapper      |
              |  - YamlShadowTableConfigSwapper           |
              |  - YamlAlgorithmConfigSwapper              |
              |                                           |
              |  Defaults:                                |
              |  - Auto-assign single data source         |
              |  - Auto-assign default algorithm          |
              +---------------------+---------------------+
                                    |
                                    v
              +-------------------------------------------+
              |     ShadowRuleConfiguration               |
              |  +-------------------------------------+  |
              |  | dataSources: Collection<             |  |
              |  |   ShadowDataSourceConfiguration>    |  |
              |  | tables: Map<String,                  |  |
              |  |   ShadowTableConfiguration>         |  |
              |  | shadowAlgorithms: Map<String,        |  |
              |  |   AlgorithmConfiguration>            |  |
              |  | defaultShadowAlgorithmName: String   |  |
              |  +-------------------------------------+  |
              +---------------------+---------------------+
                                    |
                         Step 4: Validate
                                    |
                                    v
              +-------------------------------------------+
              |  ShadowRuleConfigurationChecker           |
              |  .check()                                 |
              |                                           |
              |  1. checkShadowAlgorithms()               |
              |     -> TypedSPILoader.checkService()      |
              |  2. checkDefaultShadowAlgorithm()         |
              |     -> Must be SQL_HINT type              |
              |  3. checkDataSources()                    |
              |     -> Production & shadow must exist     |
              |  4. checkShadowTableDataSourcesRefs()     |
              |     -> Table DS refs must match declared  |
              |  5. checkShadowTableAlgorithmsRefs()      |
              |     -> Table algo refs must match declared|
              +---------------------+---------------------+
                                    |
                         Step 5: Build
                                    |
                                    v
              +-------------------------------------------+
              |  ShadowRuleBuilder.build()                |
              |  -> new ShadowRule(ruleConfig)            |
              +---------------------+---------------------+
                                    |
                         Step 6: Initialize
                                    |
                                    v
              +-------------------------------------------+
              |           ShadowRule                      |
              |  +-------------------------------------+  |
              |  | shadowAlgorithms:                    |  |
              |  |   Map<String, ShadowAlgorithm>      |  |
              |  | defaultShadowAlgorithm:             |  |
              |  |   ShadowAlgorithm (nullable)        |  |
              |  | dataSourceRules:                     |  |
              |  |   CaseInsensitiveMap<String,         |  |
              |  |     ShadowDataSourceRule>            |  |
              |  | tableRules:                          |  |
              |  |   CaseInsensitiveMap<String,         |  |
              |  |     ShadowTableRule>                 |  |
              |  +-------------------------------------+  |
              |                                           |
              |  ShadowTableRule internals:               |
              |  - hintShadowAlgorithmNames               |
              |  - columnShadowAlgorithmNames             |
              |    (EnumMap<ShadowOperationType,          |
              |     Collection<ShadowAlgorithmNameRule>>) |
              +-------------------------------------------+
                                    |
                          Ready for Query Routing
                                    |
                                    v
              +-------------------------------------------+
              |  Dynamic Change Processors (Hot Reload)   |
              |                                           |
              |  - ShadowAlgorithmChangedProcessor        |
              |  - DefaultShadowAlgorithmNameChanged-     |
              |    Processor                              |
              |  - ShadowDataSourceChangedProcessor       |
              |  - ShadowTableChangedProcessor            |
              +-------------------------------------------+

Source File Reference

File Lines Layer Purpose
ShadowRuleConfiguration.java 47 API Top-level configuration class with data sources, tables, algorithms, and default algorithm name
ShadowDataSourceConfiguration.java 35 API Immutable data source mapping: name, production data source, shadow data source
ShadowTableConfiguration.java 35 API Immutable table config: data source names and shadow algorithm names
YamlShadowRuleConfiguration.java 58 Core YAML representation with @RuleNodeTupleEntity and @RuleNodeTupleField annotations
YamlShadowDataSourceConfiguration.java 34 Core YAML data source config with production and shadow data source name fields
YamlShadowTableConfiguration.java 37 Core YAML table config with data source names and shadow algorithm names collections
YamlShadowRuleConfigurationSwapper.java 141 Core Bidirectional YAML-to-Java conversion with default assignment logic
YamlShadowDataSourceConfigurationSwapper.java 41 Core Data source config swapper (Java-to-YAML only; YAML-to-Java handled inline)
YamlShadowTableConfigurationSwapper.java 43 Core Table config bidirectional swapper
ShadowRuleConfigurationChecker.java 112 Core Five-stage validation pipeline for configuration completeness and correctness
ShadowRuleConfigurationEmptyChecker.java 37 Core Empty configuration detection (data sources or tables empty)
ShadowRuleBuilder.java 51 Core SPI builder creating ShadowRule from validated configuration
ShadowRule.java 253 Core Central runtime rule object managing algorithms, table rules, and data source mappings
ShadowTableRule.java 69 Core Per-table rule classifying algorithms into hint and column categories
ShadowDataSourceRule.java 33 Core Production-to-shadow data source pair
ShadowAlgorithmNameRule.java 33 Core Associates a shadow column name with an algorithm name
DefaultShadowAlgorithmNameChangedProcessor.java 55 Core Dynamic change processor for default shadow algorithm name
ShadowAlgorithmChangedProcessor.java 51 Core Dynamic change processor for shadow algorithm entries
ShadowDataSourceChangedProcessor.java 61 Core Dynamic change processor for data source mappings
ShadowTableChangedProcessor.java 59 Core Dynamic change processor for table configurations

GitHub URL

Workflow Repository

Retrieved from "https://leeroopedia.com/index.php?title=Workflow:Apache_Shardingsphere_Shadow_Rule_Configuration&oldid=1646"