Jump to content

Connect Leeroopedia MCP: Equip your AI agents to search best practices, build plans, verify code, diagnose failures, and look up hyperparameter defaults.

Workflow:Apache Shardingsphere Metadata DDL Refresh

From Leeroopedia


Attribute Value
Workflow Name Metadata DDL Refresh
Project Apache ShardingSphere
Domains Distributed_Database, Metadata_Management, DDL_Processing
Knowledge Sources Repo, Doc
Last Updated 2026-02-10 12:00 GMT

Overview

The Metadata DDL Refresh workflow governs how Apache ShardingSphere propagates schema changes triggered by DDL statements (CREATE, ALTER, DROP, and RENAME on tables, views, indexes, and schemas) across all nodes in a distributed cluster. When a DDL statement executes against the actual underlying database, ShardingSphere must detect the change, reload the affected metadata from the real database, persist the updated metadata to its governance repository with proper versioning, and notify all other cluster nodes so they update their local metadata caches. This ensures that every proxy or JDBC driver instance in the cluster maintains a consistent, up-to-date view of the database schema at all times.

The workflow is driven by two parallel refresh engines. The PushDownMetaDataRefreshEngine handles all DDL object types (tables, views, indexes, and schemas) by dispatching to type-specific refresher implementations loaded via the SPI (Service Provider Interface) mechanism. The FederationMetaDataRefreshEngine handles an additional layer of refresh specifically for view DDL statements, persisting view definitions used by the federation query engine (Apache Calcite-based cross-database queries). Both engines resolve schema and table names through dedicated utility classes that account for database-type-specific identifier formatting and quoting rules.

Description

Supported DDL Statement Types

The PushDownMetaDataRefreshEngine maintains a static set of 13 supported SQL statement classes that trigger metadata refresh:

Category Statement Types Refresher Class
Table CreateTableStatement, AlterTableStatement, DropTableStatement, RenameTableStatement CreateTablePushDownMetaDataRefresher, AlterTablePushDownMetaDataRefresher, DropTablePushDownMetaDataRefresher, RenameTablePushDownMetaDataRefresher
View CreateViewStatement, AlterViewStatement, DropViewStatement CreateViewPushDownMetaDataRefresher, AlterViewPushDownMetaDataRefresher, DropViewPushDownMetaDataRefresher
Index CreateIndexStatement, AlterIndexStatement, DropIndexStatement CreateIndexPushDownMetaDataRefresher, AlterIndexPushDownMetaDataRefresher, DropIndexPushDownMetaDataRefresher
Schema CreateSchemaStatement, AlterSchemaStatement, DropSchemaStatement CreateSchemaPushDownMetaDataRefresher, AlterSchemaPushDownMetaDataRefresher, DropSchemaPushDownMetaDataRefresher

The FederationMetaDataRefreshEngine supports a subset of 3 statement types limited to views: CreateViewStatement, AlterViewStatement, and DropViewStatement. These are handled by CreateViewFederationMetaDataRefresher, AlterViewFederationMetaDataRefresher, and DropViewFederationMetaDataRefresher.

SPI-Based Refresher Resolution

Both engines use the TypedSPILoader to locate the appropriate refresher implementation at runtime. The resolution follows a two-step lookup strategy:

  1. First, the engine attempts to find a refresher registered for the exact SQL statement class.
  2. If no match is found, it falls back to searching for a refresher registered for the statement's superclass.

This fallback mechanism allows database-dialect-specific statement subclasses to be handled by a common refresher registered against the base statement type.

Single Table Detection

The TableRefreshUtils.isSingleTable() method determines whether a table is a single table (not managed by any distributed rule such as sharding). When a table is identified as a single table, the refresher registers it with the MutableDataNodeRuleAttribute so that the single table rule tracks its data source and schema location. This ensures that even non-sharded tables are properly managed within the metadata layer.

Name Resolution

Two utility classes handle name resolution:

  • SchemaRefreshUtils resolves the schema name from the SQL statement context. If the statement does not explicitly specify a schema, it falls back to the default schema name for the database type (e.g., public for PostgreSQL, the database name for MySQL). The result is always lowercased.
  • TableRefreshUtils resolves table names by checking the identifier's quote character. If the identifier is unquoted, the name is formatted according to the database type's identifier pattern rules (e.g., case folding). Quoted identifiers are preserved as-is.

Metadata Persistence

The DatabaseMetaDataPersistFacade coordinates all metadata persistence operations. It delegates to specialized services:

  • TableMetaDataPersistEnabledService persists table metadata with full versioning support when schema persistence is enabled.
  • TableMetaDataPersistDisabledService provides a lightweight persistence path when schema persistence is disabled.
  • ViewMetaDataPersistService persists view metadata with versioning support.
  • SchemaMetaDataPersistService manages schema-level operations (add, drop) and delegates table and view persistence to the respective services.

The facade is constructed with a persistSchemasEnabled flag that selects the appropriate table persistence strategy.

Cluster Coordination

The TableCoordinatorChangedHandler implements GlobalDataChangedEventHandler and subscribes to ADDED events on the table coordinator node path. When a cluster node persists a table change, other nodes receive the event and this handler:

  1. Parses the qualified table name (database.schema.table) from the event key.
  2. Determines the coordinator type (CREATE or DROP).
  3. For CREATE or ALTER: deserializes the YAML table content and calls alterTable() on the local metadata manager.
  4. For DROP: calls dropTable() on the local metadata manager.
  5. Triggers an asynchronous statistics refresh via StatisticsRefreshEngine.

Usage

The Metadata DDL Refresh workflow is triggered automatically whenever a DDL statement is executed through ShardingSphere (either via the JDBC driver or the Proxy). No explicit user configuration is required to enable this behavior. The refresh mechanism operates transparently as part of the SQL execution pipeline.

Prerequisites

  • ShardingSphere must be running in Cluster Mode (using ZooKeeper or etcd as the governance registry) for cross-node metadata synchronization.
  • In Standalone Mode, the push-down refresh still operates locally, but there is no cluster coordination.
  • The underlying database must be accessible so that GenericSchemaBuilder can reload table metadata after DDL execution.

DDL Statements That Trigger Refresh

Any of the following SQL statements trigger the workflow when executed through ShardingSphere: 

-- Table operations
CREATE TABLE t_order (order_id INT PRIMARY KEY, user_id INT, status VARCHAR(50));
ALTER TABLE t_order ADD COLUMN total_amount DECIMAL(10,2);
ALTER TABLE t_order RENAME TO t_order_history;
DROP TABLE t_order;
RENAME TABLE t_order TO t_order_archive;

-- View operations
CREATE VIEW v_active_orders AS SELECT * FROM t_order WHERE status = 'ACTIVE';
ALTER VIEW v_active_orders AS SELECT * FROM t_order WHERE status = 'PENDING';
DROP VIEW v_active_orders;

-- Index operations
CREATE INDEX idx_user_id ON t_order (user_id);
ALTER INDEX idx_user_id RENAME TO idx_order_user;
DROP INDEX idx_user_id;

-- Schema operations
CREATE SCHEMA order_schema;
ALTER SCHEMA order_schema RENAME TO archive_schema;
DROP SCHEMA order_schema;

Execution Steps

Step 1: DDL Statement Execution on Actual Database

The DDL statement is parsed, routed, and executed against the actual underlying database(s). ShardingSphere's SQL execution engine pushes the DDL down to the appropriate storage unit(s) based on the configured routing rules. For sharded tables, the DDL may be executed on multiple data sources. The route units collected during execution identify which logical data source names and actual data source names were involved.

Step 2: PushDownMetaDataRefreshEngine Checks if Refresh is Needed

After DDL execution completes, the PushDownMetaDataRefreshEngine is invoked with the SQLStatementContext. The engine's isNeedRefresh() method checks whether the executed SQL statement's class is contained in the static SUPPORTED_REFRESH_TYPES set, which includes all 13 supported DDL statement types (CreateTableStatement, AlterTableStatement, DropTableStatement, RenameTableStatement, CreateViewStatement, AlterViewStatement, DropViewStatement, CreateIndexStatement, AlterIndexStatement, DropIndexStatement, CreateSchemaStatement, AlterSchemaStatement, DropSchemaStatement). If the statement type is not in this set, no refresh occurs and execution continues without metadata changes.

Step 3: Appropriate PushDownMetaDataRefresher Loaded via SPI

When refresh is needed, the engine calls refresh() which resolves the correct PushDownMetaDataRefresher implementation using TypedSPILoader.findService(). The lookup first tries the exact statement class, then falls back to the statement's superclass. The engine also resolves the schema name via SchemaRefreshUtils.getSchemaName() and determines the database type from the route units' storage units. It extracts the first logical data source name from the route units and passes all parameters to the resolved refresher's refresh() method.

Step 4: Table/View/Index/Schema Metadata Reloaded from Actual Database

The type-specific refresher performs the actual metadata reload. The behavior varies by DDL type:

  • CREATE TABLE: Uses GenericSchemaBuilder.build() to connect to the actual database and introspect the newly created table's columns, indexes, and constraints. If the table is a single table (not distributed), it registers the table with MutableDataNodeRuleAttribute.
  • ALTER TABLE: Reloads the table metadata. If the ALTER includes a RENAME clause, it loads metadata for the new table name and marks the old name for deletion.
  • DROP TABLE: Collects all table names from the DROP statement without reloading metadata since the tables no longer exist.
  • RENAME TABLE: Iterates through all rename definitions, copies the existing table's columns, indexes, and constraints to a new ShardingSphereTable with the target name, and marks the original for deletion.
  • CREATE/ALTER VIEW: Loads view metadata similar to table loading, using GenericSchemaBuilder to introspect the view definition.
  • DROP VIEW: Collects view names for deletion without reloading.
  • CREATE INDEX: Retrieves the existing table metadata and adds the new index to a copy of the table object.
  • ALTER/DROP INDEX: Reloads or modifies the parent table's index collection accordingly.
  • CREATE/ALTER/DROP SCHEMA: Manages schema-level registration or removal without table-level introspection.

Step 5: Metadata Persisted to Repository with Versioning

The refresher calls methods on MetaDataManagerPersistService such as createTable(), alterTables(), dropTables(), createSchema(), dropSchema(), or alterViews(). These methods delegate to the DatabaseMetaDataPersistFacade, which persists the metadata to the governance repository (ZooKeeper or etcd). When persistSchemasEnabled is true, the TableMetaDataPersistEnabledService writes table metadata with version tracking through VersionPersistService. When disabled, TableMetaDataPersistDisabledService writes without versioning. View metadata is always persisted with versioning via ViewMetaDataPersistService.

Step 6: Federation Metadata Refresh for Views

For view DDL statements (CREATE VIEW, ALTER VIEW, DROP VIEW), the FederationMetaDataRefreshEngine is additionally triggered after the push-down refresh. Its isNeedRefresh() checks whether the statement is one of the three view statement types. If so, it loads the appropriate FederationMetaDataRefresher via SPI and invokes it. The federation refreshers persist view definitions (the SQL text of the view) through MetaDataManagerPersistService.alterViews() or dropViews(). This separate step ensures that the federation query engine (Apache Calcite) has the view definitions it needs for cross-database query optimization and execution.

Step 7: Cluster Nodes Receive Change Events and Update Local Metadata

When metadata is persisted to the governance repository, the repository's watch mechanism (e.g., ZooKeeper watchers) generates change events. On each cluster node, the TableCoordinatorChangedHandler receives ADDED events on the table coordinator node path. The handler parses the qualified table name (in the format database.schema.table) and the coordinator type (CREATE or DROP) from the event key and value. For CREATE operations, it deserializes the YAML-encoded ShardingSphereTable and calls alterTable() on the local DatabaseMetaDataManager to update the in-memory metadata. For DROP operations, it calls dropTable() to remove the table from local metadata. After either operation, it triggers an asynchronous statistics refresh via StatisticsRefreshEngine to keep query optimization statistics current.

Execution Diagram

                          DDL Statement Executed
                          (CREATE/ALTER/DROP/RENAME)
                                   |
                                   v
                    +------------------------------+
                    | SQL Execution Engine pushes   |
                    | DDL to actual database(s)     |
                    +------------------------------+
                                   |
                                   v
                    +------------------------------+
                    | PushDownMetaDataRefreshEngine |
                    | .isNeedRefresh()              |
                    | Check SUPPORTED_REFRESH_TYPES |
                    +------------------------------+
                            |              |
                       [Not needed]    [Needed]
                            |              |
                            v              v
                         (done)   TypedSPILoader resolves
                                  PushDownMetaDataRefresher
                                           |
                     +---------------------+---------------------+
                     |                     |                     |
                     v                     v                     v
              Table Refreshers      View Refreshers       Index/Schema
              - CreateTable         - CreateView          Refreshers
              - AlterTable          - AlterView           - CreateIndex
              - DropTable           - DropView            - AlterIndex
              - RenameTable                               - DropIndex
                     |                     |              - Create/Alter/
                     |                     |                DropSchema
                     v                     v                     |
              +------------------+  +------------------+         |
              | GenericSchema    |  | GenericSchema     |        |
              | Builder.build()  |  | Builder.build()   |        |
              | (reload from DB) |  | (reload from DB)  |        |
              +------------------+  +------------------+         |
                     |                     |                     |
                     +----------+----------+---------------------+
                                |
                                v
                 +-------------------------------+
                 | MetaDataManagerPersistService  |
                 | createTable / alterTables /    |
                 | dropTables / createSchema /    |
                 | dropSchema / alterViews        |
                 +-------------------------------+
                                |
               +----------------+----------------+
               |                                 |
               v                                 v
  +-------------------------+     +-----------------------------+
  | DatabaseMetaDataPersist |     | FederationMetaDataRefresh   |
  | Facade                  |     | Engine (views only)         |
  | - TableMetaDataPersist  |     | - CreateViewFederation      |
  |   EnabledService (ver.) |     |   MetaDataRefresher         |
  | - TableMetaDataPersist  |     | - AlterViewFederation       |
  |   DisabledService       |     |   MetaDataRefresher         |
  | - ViewMetaDataPersist   |     | - DropViewFederation        |
  |   Service               |     |   MetaDataRefresher         |
  +-------------------------+     +-----------------------------+
               |                                 |
               +----------------+----------------+
                                |
                                v
                 +-------------------------------+
                 | Governance Repository          |
                 | (ZooKeeper / etcd)             |
                 | Persisted with versioning      |
                 +-------------------------------+
                                |
                     (Watch events fired)
                                |
                                v
                 +-------------------------------+
                 | TableCoordinatorChanged        |
                 | Handler (each cluster node)    |
                 | - Parse database.schema.table  |
                 | - CREATE -> alterTable()       |
                 | - DROP   -> dropTable()        |
                 +-------------------------------+
                                |
                                v
                 +-------------------------------+
                 | Local DatabaseMetaDataManager  |
                 | updates in-memory metadata     |
                 +-------------------------------+
                                |
                                v
                 +-------------------------------+
                 | StatisticsRefreshEngine        |
                 | .asyncRefresh()                |
                 +-------------------------------+

Source File Reference

File Lines Description
PushDownMetaDataRefreshEngine.java 103 Main engine that checks if DDL refresh is needed and dispatches to the appropriate SPI-loaded refresher
PushDownMetaDataRefresher.java 55 SPI interface defining the contract for push-down metadata refreshers, parameterized by SQL statement type
CreateTablePushDownMetaDataRefresher.java 67 Handles CREATE TABLE by reloading table metadata via GenericSchemaBuilder and persisting it
AlterTablePushDownMetaDataRefresher.java 79 Handles ALTER TABLE including rename scenarios, reloading and persisting updated table metadata
DropTablePushDownMetaDataRefresher.java 46 Handles DROP TABLE by collecting table names and issuing drop persistence calls
RenameTablePushDownMetaDataRefresher.java 57 Handles RENAME TABLE by copying metadata to the new name and dropping the old entry
CreateViewPushDownMetaDataRefresher.java 65 Handles CREATE VIEW by loading view metadata from the actual database
AlterViewPushDownMetaDataRefresher.java ~65 Handles ALTER VIEW by reloading the view metadata
DropViewPushDownMetaDataRefresher.java ~45 Handles DROP VIEW by issuing drop persistence calls
CreateIndexPushDownMetaDataRefresher.java 66 Handles CREATE INDEX by adding the new index to the parent table's metadata
AlterIndexPushDownMetaDataRefresher.java ~65 Handles ALTER INDEX (e.g., rename) on the parent table metadata
DropIndexPushDownMetaDataRefresher.java ~55 Handles DROP INDEX by removing the index from the parent table's metadata
CreateSchemaPushDownMetaDataRefresher.java 49 Handles CREATE SCHEMA by persisting a new empty schema entry
AlterSchemaPushDownMetaDataRefresher.java ~55 Handles ALTER SCHEMA (rename) by renaming the schema in persistence
DropSchemaPushDownMetaDataRefresher.java ~45 Handles DROP SCHEMA by removing the schema from persistence
FederationMetaDataRefreshEngine.java 77 Engine for federation-level view metadata refresh, triggered alongside push-down refresh for view DDL
FederationMetaDataRefresher.java 48 SPI interface for federation metadata refreshers, parameterized by SQL statement type
CreateViewFederationMetaDataRefresher.java 46 Persists view definition text for the federation query engine
AlterViewFederationMetaDataRefresher.java ~46 Updates view definition text in federation metadata
DropViewFederationMetaDataRefresher.java ~40 Removes view definition from federation metadata
SchemaRefreshUtils.java 43 Resolves schema name from SQL context with database-type-aware default fallback
TableRefreshUtils.java 121 Resolves table names with quote-aware formatting and determines single-table status
DatabaseMetaDataPersistFacade.java 167 Facade coordinating metadata persistence through table, view, and schema services
TableMetaDataPersistEnabledService.java ~80 Table persistence with version tracking for cluster mode
TableMetaDataPersistDisabledService.java ~60 Lightweight table persistence without versioning
TableCoordinatorChangedHandler.java 103 Cluster coordination handler that receives change events and updates local metadata on each node

GitHub URL

Workflow Repository

Relevant source paths within the repository:

  • mode/core/src/main/java/org/apache/shardingsphere/mode/metadata/refresher/pushdown/
  • mode/core/src/main/java/org/apache/shardingsphere/mode/metadata/refresher/pushdown/type/table/
  • mode/core/src/main/java/org/apache/shardingsphere/mode/metadata/refresher/pushdown/type/view/
  • mode/core/src/main/java/org/apache/shardingsphere/mode/metadata/refresher/pushdown/type/index/
  • mode/core/src/main/java/org/apache/shardingsphere/mode/metadata/refresher/pushdown/type/schema/
  • mode/core/src/main/java/org/apache/shardingsphere/mode/metadata/refresher/federation/
  • mode/core/src/main/java/org/apache/shardingsphere/mode/metadata/refresher/federation/type/
  • mode/core/src/main/java/org/apache/shardingsphere/mode/metadata/refresher/util/
  • mode/core/src/main/java/org/apache/shardingsphere/mode/metadata/persist/metadata/
  • mode/type/cluster/core/src/main/java/org/apache/shardingsphere/mode/manager/cluster/dispatch/handler/global/state/coordinator/
Retrieved from "https://leeroopedia.com/index.php?title=Workflow:Apache_Shardingsphere_Metadata_DDL_Refresh&oldid=1556"