Bidirectional Replication

TiCDC supports bidirectional replication (BDR) among two TiDB clusters. Based on this feature, you can create a multi-active TiDB solution using TiCDC.

This section describes how to use bidirectional replication taking two TiDB clusters as an example.

Deploy bidirectional replication

TiCDC only replicates incremental data changes that occur after a specified timestamp to the downstream cluster. Before starting the bidirectional replication, you need to take the following steps:

1. (Optional) According to your needs, import the data of the two TiDB clusters into each other using the data export tool Dumpling and data import tool TiDB Lightning.

2. Deploy two TiCDC clusters between the two TiDB clusters. The cluster topology is as follows. The arrows in the diagram indicate the directions of data flow.

   

3. Specify the starting time point of data replication for the upstream and downstream clusters.

   1. Check the time point of the upstream and downstream clusters. In the case of two TiDB clusters, make sure that data in the two clusters are consistent at certain time points. For example, the data of TiDB 1 at ts=1 and the data of TiDB 2 at ts=2 are consistent.

   2. When you create the changefeed, set the --start-ts of the changefeed for the upstream cluster to the corresponding tso. That is, if the upstream cluster is TiDB 1, set --start-ts=1; if the downstream cluster is TiDB 2, set --start-ts=2.

4. In the configuration file specified by the --config parameter, add the following configuration:

   # Whether to enable the bidirectional replication mode
   bdr-mode = true
   

After the configuration takes effect, the clusters can perform bidirectional replication.

DDL types

Starting from v7.6.0, to support DDL replication as much as possible in bidirectional replication, TiDB divides the DDLs that TiCDC originally supports into two types: replicable DDLs and non-replicable DDLs, according to the impact of DDLs on the business.

Replicable DDLs

Replicable DDLs are the DDLs that can be directly executed and replicated to other TiDB clusters in bidirectional replication.

Replicable DDLs include:

• ALTER TABLE ... ADD COLUMN: the column can be null, or has not null and default value at the same time
• ALTER TABLE ... ADD INDEX (non-unique)
• ALTER TABLE ... ADD PARTITION
• ALTER TABLE ... ALTER COLUMN DROP DEFAULT
• ALTER TABLE ... ALTER COLUMN SET DEFAULT
• ALTER TABLE ... COMMENT=...
• ALTER TABLE ... DROP PRIMARY KEY
• ALTER TABLE ... MODIFY COLUMN: you can only modify the default value and comment of the column
• ALTER TABLE ... RENAME INDEX
• ALTER TABLE ... ALTER INDEX ... INVISIBLE
• ALTER TABLE ... ALTER INDEX ... VISIBLE
• ALTER TABLE REMOVE TTL
• ALTER TABLE TTL
• CREATE DATABASE
• CREATE INDEX
• CREATE TABLE
• CREATE VIEW
• DROP INDEX
• DROP VIEW

Non-replicable DDLs

Non-replicable DDLs are the DDLs that have a great impact on the business, and might cause data inconsistency between clusters. Non-replicable DDLs cannot be directly replicated to other TiDB clusters in bidirectional replication through TiCDC. Non-replicable DDLs must be executed through specific operations.

Non-replicable DDLs include:

• ALTER DATABASE CHARACTER SET
• ALTER TABLE ... ADD COLUMN: the column is not null and does not have a default value
• ALTER TABLE ... ADD PRIMARY KEY
• ALTER TABLE ... ADD UNIQUE INDEX
• ALTER TABLE ... AUTO_INCREMENT=...
• ALTER TABLE ... AUTO_RANDOM_BASE=...
• ALTER TABLE ... CHARACTER SET=...
• ALTER TABLE ... DROP COLUMN
• ALTER TABLE ... DROP PARTITION
• ALTER TABLE ... EXCHANGE PARTITION
• ALTER TABLE ... MODIFY COLUMN: you can modify the attributes of the column except default value and comment
• ALTER TABLE ... REORGANIZE PARTITION
• ALTER TABLE ... TRUNCATE PARTITION
• DROP DATABASE
• DROP TABLE
• RECOVER TABLE
• RENAME TABLE
• TRUNCATE TABLE

DDL replication

To solve the problem of replicable DDLs and non-replicable DDLs, TiDB introduces the following BDR roles:

• PRIMARY: You can execute replicable DDLs, but not non-replicable DDLs. Replicable DDLs executed in a PRIMARY cluster will be replicated to the downstream by TiCDC.
• SECONDARY: You cannot execute replicable DDLs or non-replicable DDLs. However, DDLs executed in a PRIMARY cluster can be replicated to a SECONDARY cluster by TiCDC.

When no BDR role is set, you can execute any DDL. However, the changefeed in BDR mode does not replicate any DDL on that cluster.

In short, in BDR mode, TiCDC only replicates replicable DDLs in the PRIMARY cluster to the downstream.

Replication scenarios of replicable DDLs

1. Choose a TiDB cluster and execute ADMIN SET BDR ROLE PRIMARY to set it as the primary cluster.

   ADMIN SET BDR ROLE PRIMARY;
   

   Query OK, 0 rows affected
   Time: 0.003s
   
   ADMIN SHOW BDR ROLE;
   +----------+
   | BDR_ROLE |
   +----------+
   | primary  |
   +----------+
   

2. On other TiDB clusters, execute ADMIN SET BDR ROLE SECONDARY to set them as the secondary clusters.

3. Execute replicable DDLs on the primary cluster. The successfully executed DDLs will be replicated to the secondary clusters by TiCDC.

Replication scenarios of non-replicable DDLs

1. Execute ADMIN UNSET BDR ROLE on all TiDB clusters to unset the BDR role.
2. Stop writing data to the tables that need to execute DDLs in all clusters.
3. Wait until all writes to the corresponding tables in all clusters are replicated to other clusters, and then manually execute all DDLs on each TiDB cluster.
4. Wait until the DDLs are completed, and then resume writing data.
5. Follow the steps in Replication scenarios of replicable DDLs to switch back to the replication scenario of replicable DDLs.

Stop bidirectional replication

After the application has stopped writing data, you can insert a special record into each cluster. By checking the two special records, you can make sure that data in two clusters are consistent.

After the check is completed, you can stop the changefeed to stop bidirectional replication, and execute ADMIN UNSET BDR ROLE on all TiDB clusters.

Limitations

• Use BDR role only in the following scenarios:

  • 1 PRIMARY cluster and n SECONDARY clusters (replication scenarios of replicable DDLs)

  • n clusters that have no BDR roles (replication scenarios in which you can manually execute non-replicable DDLs on each cluster)

    Note

    Do not set the BDR role in other scenarios, for example, setting PRIMARY, SECONDARY, and no BDR roles at the same time. If you set the BDR role incorrectly, TiDB cannot guarantee data correctness and consistency during data replication.

• Usually do not use AUTO_INCREMENT or AUTO_RANDOM to avoid data conflicts in the replicated tables. If you need to use AUTO_INCREMENT or AUTO_RANDOM, you can set different auto_increment_increment and auto_increment_offset for different clusters to ensure that different clusters can be assigned different primary keys. For example, if there are three TiDB clusters (A, B, and C) in bidirectional replication, you can set them as follows:

  • In Cluster A, set auto_increment_increment=3 and auto_increment_offset=2000

  • In Cluster B, set auto_increment_increment=3 and auto_increment_offset=2001

  • In Cluster C, set auto_increment_increment=3 and auto_increment_offset=2002

    This way, A, B, and C will not conflict with each other in the implicitly assigned AUTO_INCREMENT ID and AUTO_RANDOM ID. If you need to add a cluster in BDR mode, you need to temporarily stop writing data of the related application, set the appropriate values for auto_increment_increment and auto_increment_offset on all clusters, and then resume writing data of the related application.

• Bidirectional replication clusters cannot detect write conflicts, which might cause undefined behaviors. Therefore, you must ensure that there are no write conflicts from the application side.

• Bidirectional replication supports more than two clusters, but does not support multiple clusters in cascading mode, that is, a cyclic replication like TiDB A -> TiDB B -> TiDB C -> TiDB A. In such a topology, if one cluster fails, the whole data replication will be affected. Therefore, to enable bidirectional replication among multiple clusters, you need to connect each cluster with every other clusters, for example, TiDB A <-> TiDB B, TiDB B <-> TiDB C, TiDB C <-> TiDB A.