TiDB Data Migration Overview
TiDB Data Migration (DM) is an integrated data migration task management platform, which supports the full data migration and the incremental data replication from MySQL-compatible databases (such as MySQL, MariaDB, and Aurora MySQL) into TiDB. It can help to reduce the operation cost of data migration and simplify the troubleshooting process.
Basic features
- Compatibility with MySQL. DM is compatible with the MySQL protocol and most of the features and syntax of MySQL 5.7 and MySQL 8.0.
- Replicating DML and DDL events. It supports parsing and replicating DML and DDL events in MySQL binlog.
- Migrating and merging MySQL shards. DM supports migrating and merging multiple MySQL database instances upstream to one TiDB database downstream. It supports customizing replication rules for different migration scenarios. It can automatically detect and handle DDL changes of upstream MySQL shards, which greatly reduces the operational cost.
- Various types of filters. You can predefine event types, regular expressions, and SQL expressions to filter out MySQL binlog events during the data migration process.
- Centralized management. DM supports thousands of nodes in a cluster. It can run and manage a large number of data migration tasks concurrently.
- Optimization of the third-party Online Schema Change process. In the MySQL ecosystem, tools such as gh-ost and pt-osc are widely used. DM optimizes its change process to avoid unnecessary migration of intermediate data. For details, see online-ddl.
- High availability. DM supports data migration tasks to be scheduled freely on different nodes. The running tasks are not affected when a small number of nodes crash.
Quick installation
Run the following command to install DM:
curl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | sh
tiup install dm dmctl
Usage restrictions
Before using the DM tool, note the following restrictions:
Database version requirements
MySQL version 5.6 ~ 8.0
MariaDB version >= 10.1.2 (experimental features)
DDL syntax compatibility
Currently, TiDB is not compatible with all the DDL statements that MySQL supports. Because DM uses the TiDB parser to process DDL statements, it only supports the DDL syntax supported by the TiDB parser. For details, see MySQL Compatibility.
DM reports an error when it encounters an incompatible DDL statement. To solve this error, you need to manually handle it using dmctl, either skipping this DDL statement or replacing it with specified DDL statements. For details, see Skip or replace abnormal SQL statements.
DM does not replicate view-related DDL statements and DML statements to the downstream TiDB cluster. It is recommended that you create the view in the downstream TiDB cluster manually.
GBK character set compatibility
- DM does not support migrating
charset=GBKtables to TiDB clusters earlier than v5.4.0.
- DM does not support migrating
Binlog compatibility
- DM does not support the MySQL 8.0 new feature binlog Transaction_payload_event. Using binlog Transaction_payload_event might result in data inconsistency between upstream and downstream.
Vector data type replication
- DM does not support migrating or replicating MySQL 9.0 vector data types to TiDB.
Contributing
You are welcome to participate in the DM open sourcing project. Your contribution would be highly appreciated. For more details, see CONTRIBUTING.md.
Community support
You can learn about DM through the online documentation. If you have any questions, contact us on GitHub.
License
DM complies with the Apache 2.0 license. For more details, see LICENSE.
DM versions
Before v5.4, the DM documentation is independent of the TiDB documentation. To access these earlier versions of the DM documentation, click one of the following links: