- Introduction
- Deploy
- Deploy TiDB Cluster
- Deploy Heterogeneous Cluster
- Deploy TiFlash
- Deploy TiCDC
- Deploy TiDB Binlog
- Deploy TiDB Enterprise Edition
- Deploy Monitoring
- Secure
- Operate
- Scale TiDB Cluster
- Backup and Restore
- Use CRDs
- Back up Data to GCS Using Dumpling
- Restore Data from GCS Using TiDB Lightning
- Back up Data to S3-Compatible Storage Using Dumpling
- Restore Data from S3-Compatible Storage Using TiDB Lightning
- Back up Data to GCS Using BR
- Restore Data from GCS Using BR
- Back up Data to S3-Compatible Storage Using BR
- Restore Data from S3-Compatible Storage Using BR
- Back up Data to PV Using BR
- Restore Data from PV Using BR
- Use CRDs
- Disaster Recovery
- Reference
- Tools
- Configure
- TiDB Operator Roadmap
- Release Notes
- v1.2
- v1.1
- v1.0
- v0
Deploy and Use DM in Kubernetes
TiDB Data Migration (DM) is an integrated data migration task management platform that supports the full data migration and the incremental data replication from MySQL/MariaDB into TiDB. This document describes how to deploy DM in Kubernetes using TiDB Operator and how to migrate MySQL data to TiDB cluster using DM.
Prerequisites
- Complete deploying TiDB Operator.
- Complete deploying a TiDB Cluster.
Configure DM deployment
To configure the DM deployment, you need to configure the DMCluster
Custom Resource (CR). For the complete configurations of the DMCluster
CR, refer to the DMCluster example and API documentation. Note that you need to choose the example and API of the current TiDB Operator version.
Cluster name
Configure the cluster name by changing the metadata.name
in the DMCluster
CR.
Version
Usually, components in a cluster are in the same version. It is recommended to configure only spec.<master/worker>.baseImage
and spec.version
. If you need to deploy different versions for different components, configure spec.<master/worker>.version
.
The formats of the related parameters are as follows:
spec.version
: the format isimageTag
, such asv2.0.0-rc.2
.spec.<master/worker>.baseImage
: the format isimageName
, such aspingcap/tidb
.spec.<master/worker>.version
: the format isimageTag
, such asv2.0.0-rc.2
.
TiDB Operator only supports deploying DM 2.0 and later versions.
Cluster
Configure Discovery
To deploy the DM cluster, you need to use the Discovery service of TidbCluster and must fill in the spec.discovery.address
. The format is http://${tidb_cluster_name}-discovery.${tidb_namespace}:10261
.
apiVersion: pingcap.com/v1alpha1
kind: DMCluster
metadata:
name: ${dm_cluster_name}
namespace: ${namespace}
spec:
...
discovery:
address: "http://${tidb_cluster_name}-discovery.${tidb_namespace}:10261"
Configure DM-master
DM-master is an indispensable component of the DM cluster. You need to deploy at least three DM-master Pods if you want to achieve high availability.
You can configure DM-master parameters by spec.master.config
in DMCluster
CR. For complete DM-master configuration parameters, refer to DM-master Configuration File.
apiVersion: pingcap.com/v1alpha1
kind: DMCluster
metadata:
name: ${dm_cluster_name}
namespace: ${namespace}
spec:
version: v2.0.0-rc.2
pvReclaimPolicy: Retain
discovery:
address: "http://${tidb_cluster_name}-discovery.${tidb_namespace}:10261"
master:
baseImage: pingcap/dm
imagePullPolicy: IfNotPresent
service:
type: NodePort
# Configures masterNodePort when you need to expose the DM-master service to a fixed NodePort
# masterNodePort: 30020
replicas: 1
storageSize: "1Gi"
requests:
cpu: 1
config:
rpc-timeout: 40s
Configure DM-worker
You can configure DM-worker parameters by spec.worker.config
in DMCluster
CR. For complete DM-worker configuration parameters,refer to DM-worker Configuration File.
apiVersion: pingcap.com/v1alpha1
kind: DMCluster
metadata:
name: ${dm_cluster_name}
namespace: ${namespace}
spec:
...
worker:
baseImage: pingcap/dm
replicas: 1
storageSize: "1Gi"
requests:
cpu: 1
config:
keepalive-ttl: 15
Deploy the DM cluster
After configuring the yaml file of the DM cluster in the above steps, execute the following command to deploy the DM cluster:
kubectl apply -f ${dm_cluster_name}.yaml -n ${namespace}
If the server does not have an external network, you need to download the Docker image used by the DM cluster and upload the image to the server, and then execute docker load
to install the Docker image on the server:
Deploy a DM cluster requires the following Docker image (assuming the version of the DM cluster is v2.0.0-rc.2):
pingcap/dm:v2.0.0-rc.2
To download the image, execute the following command:
docker pull pingcap/dm:v2.0.0-rc.2 docker save -o dm-v2.0.0-rc.2.tar pingcap/dm:v2.0.0-rc.2
Upload the Docker image to the server, and execute
docker load
to install the image on the server:docker load -i dm-v2.0.0-rc.2.tar
After deploying the DM cluster, execute the following command to view the Pod status:
kubectl get po -n ${namespace} -l app.kubernetes.io/instance=${dm_cluster_name}
You can use TiDB Operator to deploy and manage multiple DM clusters in a single Kubernetes cluster by repeating the above procedure and replacing ${dm_cluster_name}
with a different name.
Different clusters can be in the same or different namespace
, which is based on your actual needs.
Access the DM cluster in Kubernetes
To access DM-master in the pod within a Kubernetes cluster, use the DM-master service domain name ${cluster_name}-dm-master.${namespace}
.
To access the DM cluster outside a Kubernetes cluster, expose the DM-master port by editing the spec.master.service
field configuration in the DMCluster
CR.
spec:
...
master:
service:
type: NodePort
You can access the DM-master service via the address of ${kubernetes_node_ip}:${node_port}
.
For more service exposure methods, refer to Access the TiDB Cluster.
Enable DM data migration tasks
You can access the DM-master service using dmctl in the following two methods:
Method #1: Attach to the DM-master or DM-worker Pod to use the built-in dmctl
in the image.
Method #2: Expose the DM-master service by accessing the DM cluster in Kubernetes and use dmctl
outside the pods to access the exposed DM-master service.
It is recommended to use Method #1 for migration. The following steps take Method #1 as an example to introduce how to start a DM data migration task.
The differences between Method #1 and Method #2 are that the file locations of source.yaml
and task.yaml
are different, and that in Method #2 you need to configure the exposed DM-master service address in the master-addr
configuration item of dmctl
.
Get into the Pod
Attach to the DM-master Pod by executing the following command:
kubectl exec -ti ${dm_cluster_name}-dm-master-0 -n ${namespace} - /bin/sh
Create data source
Write MySQL-1 related information to
source1.yaml
file, which can refer to Create data source.Configure the
from.host
in thesource1.yaml
file as the MySQL host address that the Kubernetes cluster can access internally.After you prepare the
source1.yaml
file, load the MySQL-1 data source into the DM cluster by executing the following command:/dmctl --master-addr ${dm_cluster_name}-dm-master:8261 operate-source create source1.yaml`.
For MySQL-2 and other data sources, use the same method to modify the relevant information in the configuration file and execute the same dmctl command.
Configure migration tasks
Edit task configuration file
task.yaml
, which can refer to Configure the data migration task.Configure the
target-database.host
intask.yaml
as the TiDB host address that the Kubernetes cluster can access internally. If the cluster is deployed by TiDB Operator, configure the host as${tidb_cluster_name}-tidb.${namespace}
.
Start/Check/Stop the migration tasks
Refer to the Steps 5-7 in Migrate Data Using DM and fill in the master-addr as ${dm_cluster_name}-dm-master:8261
.