Sign InTry Free

Migrate from Amazon Aurora MySQL Using TiDB Lightning

This document introduces how to migrate full data from Amazon Aurora MySQL to TiDB using TiDB Lightning.

Step 1: Export full data from Aurora to Amazon S3

Refer to AWS Documentation - Exporting DB snapshot data to Amazon S3 to export the snapshot data of Aurora to Amazon S3.

Step 2: Deploy TiDB Lightning

For detailed deployment methods, see Deploy TiDB Lightning.

Step 3: Configure the data source of TiDB Lightning

Based on different deployment methods, edit the tidb-lighting.toml configuration file as follows:

  1. Configure data-source-dir under [mydumper] as the S3 Bucket path of exported data in step 1.

    # Data source directory
    data-source-dir = "s3://bucket-name/data-path"
  2. Configure the target TiDB cluster as follows:

    # The target cluster information. Fill in one address of tidb-server.
    host = ""
    port = 4000
    user = "root"
    password = ""
    # The PD address of the cluster.
    pd-addr = ""
  3. Configure the backend mode:

    # Uses Local-backend.
    backend = "local"
    # The storage path of local temporary files. Ensure that the corresponding directory does not exist or is empty and that the disk capacity is greater than the size of the dataset to be imported.
    sorted-kv-dir = "/path/to/local-temp-dir"
  4. Configure the file routing.

    no-schema = true
    # Uses single quoted strings to avoid escaping.
    pattern = '(?i)^(?:[^/]*/)*([a-z0-9_]+)\.([a-z0-9_]+)/(?:[^/]*/)*(?:[a-z0-9\-_.]+\.(parquet))$'
    schema = '$1'
    table = '$2'
    type = '$3'

For other configurations, see TiDB Lightning Configuration.

Step 4: Create table schema

Because the snapshot data exported from Aurora to S3 does not contain the SQL statement file used to create database tables, you need to manually export and import the table creation statements corresponding to the database tables into TiDB. You can use Dumpling and TiDB Lightning to create all table schemas:

  1. Use Dumpling to export table schema files:

    ./dumpling --host --port 3306 --user root --password password --consistency none --no-data --output ./schema --filter "mydb.*"
  2. Use TiDB Lightning to create table schemas:

    ./tidb-lightning -config tidb-lightning.toml -d ./schema -no-schema=false

    In this example, TiDB Lightning is only used to create table schemas, so you need to execute the above command quickly. At a regular speed, ten table creation statements can be executed in one second.

Step 5: Run TiDB Lightning to import data

Run TiDB Lightning to start the import operation. If you start TiDB Lightning by using nohup directly in the command line, the program might exit because of the SIGHUP signal. Therefore, it is recommended to write nohup in a script. For example:

# !/bin/bash
export AWS_ACCESS_KEY_ID=${AccessKey}
export AWS_SECRET_ACCESS_KEY=${SecretKey}
nohup ./tidb-lightning -config tidb-lightning.toml > nohup.out &

When the import operation is started, view the progress by the following two ways:

  • grep the keyword progress in logs, which is updated every 5 minutes by default.
  • Access the monitoring dashboard. See Monitor TiDB Lightning for details.
Download PDFRequest docs changesAsk questions on TiDB Forum
Was this page helpful?
TiDB Cloud
Get Demo
© 2023 PingCAP. All Rights Reserved.