Changefeed DDL Replication
This document describes the rules and special cases of DDL replication in TiCDC.
DDL allow list
Currently, TiCDC uses an allow list to determine whether to replicate a DDL statement. Only the DDL statements in the allow list are replicated to the downstream. The DDL statements not in the allow list are not replicated.
In addition, TiCDC determines whether to replicate a DDL statement to the downstream based on whether the table has a valid index and whether the configuration item force-replicate is set to true. When force-replicate=true, the replication task attempts to forcibly replicate tables without a valid index.
The following is the allow list of DDL statements supported by TiCDC. The abbreviations in the table:
- Y: Replication to the downstream is supported in this condition.
- N: Replication to the downstream is not supported in this condition.
DDL replication considerations
Asynchronous execution of ADD INDEX and CREATE INDEX DDLs
Although TiCDC usually replicates DDL statements in order, when the downstream is TiDB, TiCDC executes DDL operations for creating and adding indexes asynchronously. This means that, after replicating ADD INDEX and CREATE INDEX DDLs to the downstream TiDB cluster for execution, TiCDC returns immediately without waiting for DDL execution to complete. This minimizes the impact on changefeed replication latency and avoids blocking subsequent DML executions.
During the execution of the ADD INDEX or CREATE INDEX DDL operation in the downstream, when TiCDC executes the next DDL operation of the same table, this DDL operation might be blocked in the queueing state for a long time. This can cause TiCDC to repeatedly execute this DDL operation, and if retries take too long, it might lead to replication task failure. Starting from v8.4.0, if TiCDC has the SUPER permission of the downstream database, it periodically runs ADMIN SHOW DDL JOBS to check the status of asynchronously executed DDL tasks. TiCDC will wait for index creation to complete before proceeding with replication. Although this might increase replication latency, it avoids replication task failure.
DDL replication considerations for renaming tables
TiCDC determines whether to replicate RENAME TABLE DDLs based on the database and table names before and after renaming and the filter rules. The following examples assume that the tables meet the replication requirements and that no additional DDL event filter rules are configured.
Rename a single table in a DDL statement
If a DDL statement renames a single table, TiCDC only replicates the DDL statement when the old table name matches the filter rule. The following is an example.
Assume that the configuration file of your changefeed is as follows:
[filter]
rules = ['test.t*']
TiCDC processes this type of DDL as follows:
These rules apply to both the classic and new architectures. If the old table name matches the filter rule but the new table name does not, TiCDC replicates the RENAME TABLE DDL and then stops replicating subsequent data changes for that table. If the old table name does not match but the new table name matches, TiCDC reports the CDC:ErrSyncRenameTableFailed error because replicating only the rename cannot backfill data that was not previously replicated for that table.
Rename multiple tables in a DDL statement
When a DDL statement renames multiple tables, the new architecture evaluates each rename step according to the single-table rename rules:
- If the old table name matches the filter rule, TiCDC replicates the corresponding rename step.
- If neither the old nor the new table name matches, TiCDC ignores that step.
- If the old table name does not match but the new table name does, the changefeed reports an error and replication stops.
In addition, the classic architecture requires the target database of each replicated rename step to be within the replication scope. It also does not support RENAME TABLE statements that swap table names.
Assume that the configuration file of your changefeed is as follows:
[filter]
rules = ['test.t*']
TiCDC processes this type of DDL as follows:
In the new architecture, TiCDC coordinates the replication progress of all tables involved in a multi-table rename, but reconstructs the downstream SQL as multiple single-table RENAME TABLE statements. The MySQL sink executes multiple sub-statements separated by semicolons, while the Kafka and Pulsar sinks split them into single-table DDL messages. Therefore, downstream execution does not guarantee the atomicity of a single upstream multi-table rename statement. When using a temporary table name to swap table names, keep all mutually dependent steps and avoid removing any of them through filter rules.
DDL statement considerations
When executing cross-database DDL statements (such as CREATE TABLE db1.t1 LIKE t2) in the upstream, it is recommended that you explicitly specify all relevant database names in DDL statements (such as CREATE TABLE db1.t1 LIKE db2.t2). Otherwise, cross-database DDL statements might not be executed correctly in the downstream due to the lack of database name information.
Handling of DDL SQL text
TiCDC parses, formats, or rebuilds DDL SQL. Therefore, the SQL text replicated downstream is not guaranteed to exactly match the original upstream statement. Both the classic and new architectures normalize keyword case, identifier quotes, spaces, and some synonymous syntax, and both also handle TiDB-specific syntax for compatibility. DDLs such as DROP TABLE, DROP VIEW, and EXCHANGE PARTITION all have SQL rebuilding logic in both architectures.
The new architecture might also modify DDL SQL in the following ways. Except for MySQL sink rewrites specifically noted, these changes also affect the SQL field in message protocols that support DDL. The exact output format might vary by TiCDC version.
For example, assume that the current database is db, table routing is not enabled, and the following rename events are not filtered. The outputs for single-table rename are as follows. Statement-ending semicolons are omitted in the table.
For RENAME TABLE a TO b, c TO d, when output as single-table events, the first SQL in the classic architecture is RENAME TABLE `db`.`a` TO `db`.`b` , and in the new architecture it is RENAME TABLE `db`.`a` TO `db`.`b`;. The trailing semicolon comes from the multi-table DDL splitting in the new architecture.
When developing downstream consumers or comparing the outputs of the classic and new architectures, parse the SQL and use the DDL type and database and table metadata to assess its meaning instead of comparing SQL text directly. Database names, table names, index names, and their letter case still need to be validated according to the specific scenario to avoid overlooking name resolution or table routing errors.
Notes on using event filter rules to filter DDL events
If a filtered DDL statement involves table creation or deletion, TiCDC only filters out the DDL statement without affecting the replication behavior of DML statements. The following is an example.
Assume that the configuration file of your changefeed is as follows:
[filter]
rules = ['test.t*']
[[filter.event-filters]]
matcher = ["test.t1"] # This filter rule applies only to the t1 table in the test database.
ignore-event = ["create table", "drop table", "truncate table", "rename table"]
When using ignore-sql to filter multi-table rename operations, note that the matching input differs between the classic and new architectures. In the classic architecture, ignore-sql is matched against each SQL statement after splitting the operation into single-table DDL events. In the new architecture, ignore-sql is matched against the reconstructed SQL for the entire multi-table rename operation when event filter rules are applied to individual tables. If the matcher of the same event filter rule matches multiple tables, ignore-sql matching one rename step might also filter other rename steps covered by that rule. Before switching architectures, check the actual matching results of such filter rules.