TiCDC Table Routing New in v8.5.7

TiCDC table routing enables you to map upstream tables to specific downstream database names or table names through changefeed configuration. This feature applies only to the TiCDC new architecture and is not supported by the TiCDC classic architecture.

Table routing modifies only the database and table names that TiCDC outputs to the downstream. It does not modify row data, column names, table schema, table filter rules, topic dispatch rules, partition dispatch rules, or column selector rules.

Usage scenarios

You can configure table routing in the following scenarios:

• Replicating sales.orders to archive.sales_orders, or to a table that follows other downstream naming conventions.
• Replicating multiple source databases to the same target database while keeping target table names unique to distinguish the source database, for example, replicating tenant_001.orders to tenant_mirror.tenant_001_orders.
• Building changefeeds for migration, disaster recovery, archiving, or shadow traffic to avoid writing to downstream objects with the same names as upstream ones.
• Exposing stable database and table names to MQ consumer applications or storage services.

Configure table routing

Before configuring table routing, enable the TiCDC new architecture. For more information, see newarch.

The following example routes sales.orders to archive.sales_orders:

[sink]
[[sink.dispatchers]]
matcher = ["sales.orders"]
target-schema = "archive"
target-table = "{schema}_{table}"

After the changefeed starts, TiCDC writes the DML and DDL events from sales.orders to the downstream archive.sales_orders.

• For database-level DDL such as CREATE DATABASE, DROP DATABASE, and ALTER DATABASE, TiCDC must be able to determine a unique target database based on the routing rules. Otherwise, replication of that DDL fails.
• If you want to route different tables in the same upstream database to different target databases, you need to create the downstream target databases in advance and avoid executing upstream database-level DDL that requires automatic replication.

Configuration fields

To use the table routing feature, you can configure the following fields in sink.dispatchers:

The matching behavior is as follows:

• Only sink.dispatchers rules that specify target-schema or target-table are used for table routing.
• If a table matches multiple table routing rules, only the first matching rule in sink.dispatchers takes effect.
• matcher always matches upstream database and table names, not the routed target database and table names.
• The changefeed configuration item case-sensitive affects only whether the matcher in table routing is case-sensitive. It does not change the case of values expanded from {schema} and {table}. For more information, see case-sensitive.

Placeholders

You can use the following placeholders in target-schema and target-table:

The values of target-schema and target-table can contain only literal text, {schema}, and {table}. If you use an unknown placeholder such as {db}, TiCDC rejects the changefeed configuration and returns the CDC:ErrInvalidTableRoutingRule error.

Using the source table sales.orders as an example:

Examples

Route all tables in one database

The following configuration routes all tables in the sales database to the archive database and appends _bak to the target table names:

[filter]
rules = ["sales.*"]

[sink]
[[sink.dispatchers]]
matcher = ["sales.*"]
target-schema = "archive"
target-table = "{table}_bak"

Example routing results:

• sales.orders is routed to archive.orders_bak.
• sales.order_items is routed to archive.order_items_bak.

Route multiple databases to the same target database

To route multiple databases to the same target database, it is recommended to include {schema} in target-table to ensure unique target table names.

[filter]
rules = ["sales.*", "crm.*", "finance.*"]

[sink]
[[sink.dispatchers]]
matcher = ["sales.*", "crm.*", "finance.*"]
target-schema = "archive"
target-table = "{schema}_{table}"

Example routing results:

• sales.orders is routed to archive.sales_orders.
• crm.orders is routed to archive.crm_orders.
• finance.orders is routed to archive.finance_orders.

Use table routing together with Kafka sink topic and partition dispatchers

You can configure both table routing fields and existing dispatch fields in the same dispatcher rule:

[filter]
rules = ["sales.orders"]

[sink]
[[sink.dispatchers]]
matcher = ["sales.orders"]
topic = "order-events"
partition = "index-value"
target-schema = "public"
target-table = "orders"

In the preceding example, table routing modifies the database and table names exposed in downstream data to public.orders.

Table routing does not change the dispatch results of topic or partition. The topic and partition dispatchers still perform matching and dispatch calculation based on the upstream table sales.orders.

Output behavior

DDL behavior

After you enable table routing, TiCDC rewrites DDL statements so that the database and table names in structured DDL metadata remain consistent with those in the SQL text.

For example, if the following rule is configured:

[[sink.dispatchers]]
matcher = ["sales.*"]
target-schema = "archive"
target-table = "{table}_routed"

For the following upstream DDL statement:

RENAME TABLE `sales`.`temp_table` TO `sales`.`renamed_table`;

TiCDC rewrites it into the following downstream DDL statement:

RENAME TABLE `archive`.`temp_table_routed` TO `archive`.`renamed_table_routed`;

If a DDL statement contains table references and the referenced tables match table routing rules, TiCDC also rewrites the referenced table names. For example, TiCDC can rewrite table references in CREATE VIEW statements and foreign key references in ALTER TABLE statements.

For database-level DDL such as CREATE DATABASE, DROP DATABASE, and ALTER DATABASE ... CHARACTER SET/COLLATE, if the database name matches table routing rules, TiCDC rewrites the database name. If the same upstream database matches multiple table routing rules and these rules map to different target database names, TiCDC cannot determine a unique target database for that database-level DDL, and the changefeed returns a table routing error.

When creating or updating a changefeed, TiCDC checks for target table conflicts based on the tables currently in the replication scope. At runtime, TiCDC updates the conflict detection state when replicating DDL such as CREATE TABLE, RENAME TABLE, DROP TABLE, and DROP DATABASE. For a database-level DDL statement, TiCDC evaluates whether it can determine a unique target database when replicating it.

Route conflict detection

A route conflict occurs when two different upstream tables are routed to the same downstream (schema, table) within a single changefeed. TiCDC does not support merging multiple upstream tables into the same downstream table.

For example, the following configuration might cause a conflict:

[filter]
rules = ["sales.*", "crm.*"]

[sink]
[[sink.dispatchers]]
matcher = ["sales.*"]
target-schema = "archive"
target-table = "{table}"

[[sink.dispatchers]]
matcher = ["crm.*"]
target-schema = "archive"
target-table = "{table}"

If both sales.orders and crm.orders are within the replication scope, both tables are routed to archive.orders. TiCDC rejects the creation or update of the changefeed and returns the CDC:ErrTableRouteConflict error.

While the changefeed is running, if a DDL statement such as CREATE TABLE or RENAME TABLE causes two upstream tables currently within the replication scope to be routed to the same target table, the changefeed fails and returns the CDC:ErrTableRouteConflict error.

If an upstream table is dropped or renamed, TiCDC removes the routing relationship between that upstream table and the target table. After that, a new upstream table can continue to use that target table. However, at any given time, the same target table can correspond to only one upstream table being replicated by the current changefeed.

Troubleshooting

Related documentation

• CLI and Configuration Parameters of TiCDC Changefeeds
• Changefeed Log Filters
• Replicate Data to MySQL-compatible Databases
• Replicate Data to Kafka
• Replicate Data to Pulsar
• Replicate Data to Storage Services