- About DM
- What is DM?
- Basic Features
- Advanced Features
- Merge and Migrate Data from Sharded Tables
- Migrate from MySQL Databases that Use GH-ost/PT-osc
- Filter Certain Row Changes Using SQL Expressions
- DM Architecture
- Quick Start
- Cluster Upgrade
- Manage Data Source
- Manage a Data Migration Task
- Manually Handle Sharding DDL Lock
- Manage Schemas of Tables to be Migrated
- Handle Alerts
- Daily Check
- Usage Scenarios
- Performance Tuning
- Release Notes
This document introduces the configuration file of the upstream database, including a configuration file template and the description of each configuration parameter in this file.
The following is a configuration file template of the upstream database:
source-id: "mysql-replica-01" # Whether to enable GTID. enable-gtid: false # Whether to enable relay log. enable-relay: false # Since DM v2.0.2, this configuration item is deprecated. To enable the relay log feature, use the `start-relay` command instead. relay-binlog-name: "" # The file name from which DM-worker starts to pull the binlog. relay-binlog-gtid: "" # The GTID from which DM-worker starts to pull the binlog. relay-dir: "relay-dir" # The directory used to store relay log. The default value is "relay-dir". from: host: "127.0.0.1" port: 3306 user: "root" password: "ZqMLjZ2j5khNelDEfDoUhkD5aV5fIJOe0fiog9w=" # The user password of the upstream database. It is recommended to use the password encrypted with dmctl. security: # The TLS configuration of the upstream database ssl-ca: "/path/to/ca.pem" ssl-cert: "/path/to/cert.pem" ssl-key: "/path/to/key.pem" # purge: # interval: 3600 # expires: 0 # remain-space: 15 # checker: # check-enable: true # backoff-rollback: 5m0s # backoff-max: 5m0s # The maximum value of backoff, should be larger than 1s # Configure binlog event filters. New in DM v2.0.2 # case-sensitive: false # filters: # - schema-pattern: dmctl # table-pattern: t_1 # events:  # sql-pattern: # - alter table .* add column `aaa` int # action: Ignore
In DM v2.0.1, DO NOT set
true at the same time. Otherwise, it may cause loss of incremental data.
This section describes each configuration parameter in the configuration file.
|Represents a MySQL instance ID.|
|Determines whether to pull binlog from the upstream using GTID. The default value is |
|Determines whether to enable the relay log feature. The default value is |
|Specifies the file name from which DM-worker starts to pull the binlog. For example, |
|Specifies the GTID from which DM-worker starts to pull the binlog. For example, |
|Specifies the relay log directory.|
|Specifies the host of the upstream database.|
|Specifies the port of the upstream database.|
|Specifies the username of the upstream database.|
|Specifies the user password of the upstream database. It is recommended to use the password encrypted with dmctl.|
|Specifies the TLS config of the upstream database. The configured file paths of the certificates must be accessible to all nodes. If the configured file paths are local paths, then all the nodes in the cluster need to store a copy of the certificates in the same path of each host.|
Generally, there is no need to manually configure these parameters unless there is a large amount of relay logs and disk capacity is insufficient.
|Sets the time interval at which relay logs are regularly checked for expiration, in seconds.|
|Sets the expiration time for relay logs, in hours. The relay log that is not written by the relay processing unit, or does not need to be read by the existing data migration task will be deleted by DM if it exceeds the expiration time. If this parameter is not specified, the automatic purge is not performed.|
|Sets the minimum amount of free disk space, in gigabytes. When the available disk space is smaller than this value, DM-worker tries to delete relay logs.|
The automatic data purge strategy only takes effect when
interval is not 0 and at least one of the two configuration items
remain-space is not 0.
DM periodically checks the current task status and error message to determine if resuming the task will eliminate the error. If needed, DM automatically retries to resume the task. DM adjusts the checking interval using the exponential backoff strategy. Its behaviors can be adjusted by the following configuration.
|Whether to enable this feature.|
|If the current checking interval of backoff strategy is larger than this value and the task status is normal, DM will try to decrease the interval.|
|The maximum value of checking interval of backoff strategy, must be larger than 1 second.|
Starting from DM v2.0.2, you can configure binlog event filters in the source configuration file.
|Determines whether the filtering rules are case-sensitive. The default value is |
|Sets binlog event filtering rules. For details, see Binlog event filter parameter explanation.|