Back up Data to Azure Blob Storage Using BR
This document describes how to back up the data of a TiDB cluster on Kubernetes to Azure Blob Storage.
The backup method described in this document is implemented based on CustomResourceDefinition (CRD) in TiDB Operator. For the underlying implementation, BR is used to get the backup data of the TiDB cluster, and then send the data to Azure Blob Storage. BR stands for Backup & Restore, which is a command-line tool for distributed backup and recovery of the TiDB cluster data.
Usage scenarios
If you have the following backup needs, you can use BR to make an ad-hoc backup or scheduled full backup of the TiDB cluster data to Azure Blob Storage.
- To back up a large volume of data (more than 1 TB) at a fast speed
- To get a direct backup of data as SST files (key-value pairs)
For other backup needs, refer to Backup and Restore Overview to choose an appropriate backup method.
Ad-hoc backup
Ad-hoc backup supports both full backup and incremental backup.
To get an Ad-hoc backup, you need to create a Backup
Custom Resource (CR) object to describe the backup details. Then, TiDB Operator performs the specific backup operation based on this Backup
object. If an error occurs during the backup process, TiDB Operator does not retry, and you need to handle this error manually.
This document provides an example about how to back up the data of the demo1
TiDB cluster in the test1
Kubernetes namespace to Azure Blob Storage. The following are the detailed steps.
Step 1. Prepare for an ad-hoc backup
Download backup-rbac.yaml, and execute the following command to create the role-based access control (RBAC) resources in the
test1
namespace:kubectl apply -f backup-rbac.yaml -n test1Grant permissions to the remote storage. You can grant permissions to Azure Blob Storage by two methods. For details, refer to Azure account permissions.
For a TiDB version earlier than v4.0.8, you also need to complete the following preparation steps. For TiDB v4.0.8 or a later version, skip these preparation steps.
Make sure that you have the
SELECT
andUPDATE
privileges on themysql.tidb
table of the backup database so that theBackup
CR can adjust the GC time before and after the backup.Create
backup-demo1-tidb-secret
to store the account and password to access the TiDB cluster:kubectl create secret generic backup-demo1-tidb-secret --from-literal=password=${password} --namespace=test1
Step 2. Perform an ad-hoc backup
Depending on which method you choose to grant permissions to the remote storage when preparing for the ad-hoc backup, export your data to Azure Blob Storage by doing one of the following:
Method 1: If you grant permissions by access key, create the
Backup
CR to back up cluster data as described below:kubectl apply -f backup-azblob.yamlThe content of
backup-azblob.yaml
is as follows:--- apiVersion: pingcap.com/v1alpha1 kind: Backup metadata: name: demo1-backup-azblob namespace: test1 spec: backupType: full br: cluster: demo1 clusterNamespace: test1 # logLevel: info # statusAddr: ${status_addr} # concurrency: 4 # rateLimit: 0 # timeAgo: ${time} # checksum: true # sendCredToTikv: true # options: # - --lastbackupts=420134118382108673 # Only needed for TiDB Operator < v1.1.10 or TiDB < v4.0.8 from: host: ${tidb_host} port: ${tidb_port} user: ${tidb_user} secretName: backup-demo1-tidb-secret azblob: secretName: azblob-secret container: my-container prefix: my-folderMethod 2: If you grant permissions by Azure AD, create the
Backup
CR to back up cluster data as described below:kubectl apply -f backup-azblob.yamlThe content of
backup-azblob.yaml
is as follows:--- apiVersion: pingcap.com/v1alpha1 kind: Backup metadata: name: demo1-backup-azblob namespace: test1 spec: backupType: full serviceAccount: tidb-backup-manager br: cluster: demo1 sendCredToTikv: false clusterNamespace: test1 # logLevel: info # statusAddr: ${status_addr} # concurrency: 4 # rateLimit: 0 # timeAgo: ${time} # checksum: true # options: # - --lastbackupts=420134118382108673 # Only needed for TiDB Operator < v1.1.10 or TiDB < v4.0.8 from: host: ${tidb_host} port: ${tidb_port} user: ${tidb_user} secretName: backup-demo1-tidb-secret azblob: secretName: azblob-secret-ad container: my-container prefix: my-folder
When configuring backup-azblob.yaml
, note the following:
- Since TiDB Operator v1.1.6, if you want to back up data incrementally, you only need to specify the last backup timestamp
--lastbackupts
inspec.br.options
. For the limitations of incremental backup, refer to Use BR to Back up and Restore Data. - For more information about Azure Blob Storage configuration, refer to Azure Blob Storage fields.
- Some parameters in
.spec.br
are optional, such aslogLevel
andstatusAddr
. For more information about BR configuration, refer to BR fields. - For v4.0.8 or a later version, BR can automatically adjust
tikv_gc_life_time
. You do not need to configurespec.tikvGCLifeTime
andspec.from
fields in theBackup
CR. - For more information about the
Backup
CR fields, refer to Backup CR fields.
After you create the Backup
CR, TiDB Operator starts the backup automatically. You can view the backup status by running the following command:
kubectl get bk -n test1 -o wide
Backup CR examples
Back up data of all clusters
---
apiVersion: pingcap.com/v1alpha1
kind: Backup
metadata:
name: demo1-backup-azblob
namespace: test1
spec:
backupType: full
serviceAccount: tidb-backup-manager
br:
cluster: demo1
sendCredToTikv: false
clusterNamespace: test1
# Only needed for TiDB Operator < v1.1.10 or TiDB < v4.0.8
# from:
# host: ${tidb_host}
# port: ${tidb_port}
# user: ${tidb_user}
# secretName: backup-demo1-tidb-secret
azblob:
secretName: azblob-secret-ad
container: my-container
prefix: my-folder
Back up data of a single database
The following example backs up data of the db1
database.
---
apiVersion: pingcap.com/v1alpha1
kind: Backup
metadata:
name: demo1-backup-azblob
namespace: test1
spec:
backupType: full
serviceAccount: tidb-backup-manager
tableFilter:
- "db1.*"
br:
cluster: demo1
sendCredToTikv: false
clusterNamespace: test1
# Only needed for TiDB Operator < v1.1.10 or TiDB < v4.0.8
# from:
# host: ${tidb_host}
# port: ${tidb_port}
# user: ${tidb_user}
# secretName: backup-demo1-tidb-secret
azblob:
secretName: azblob-secret-ad
container: my-container
prefix: my-folder
Back up data of a single table
The following example backs up data of the db1.table1
table.
---
apiVersion: pingcap.com/v1alpha1
kind: Backup
metadata:
name: demo1-backup-azblob
namespace: test1
spec:
backupType: full
serviceAccount: tidb-backup-manager
tableFilter:
- "db1.table1"
br:
cluster: demo1
sendCredToTikv: false
clusterNamespace: test1
# Only needed for TiDB Operator < v1.1.10 or TiDB < v4.0.8
# from:
# host: ${tidb_host}
# port: ${tidb_port}
# user: ${tidb_user}
# secretName: backup-demo1-tidb-secret
azblob:
secretName: azblob-secret-ad
container: my-container
prefix: my-folder
Back up data of multiple tables using the table filter
The following example backs up data of the db1.table1
table and db1.table2
table.
---
apiVersion: pingcap.com/v1alpha1
kind: Backup
metadata:
name: demo1-backup-azblob
namespace: test1
spec:
backupType: full
serviceAccount: tidb-backup-manager
tableFilter:
- "db1.table1"
- "db1.table2"
# ...
br:
cluster: demo1
sendCredToTikv: false
clusterNamespace: test1
# Only needed for TiDB Operator < v1.1.10 or TiDB < v4.0.8
# from:
# host: ${tidb_host}
# port: ${tidb_port}
# user: ${tidb_user}
# secretName: backup-demo1-tidb-secret
azblob:
secretName: azblob-secret-ad
container: my-container
bucket: my-bucket
prefix: my-folder
Scheduled full backup
You can set a backup policy to perform scheduled backups of the TiDB cluster, and set a backup retention policy to avoid excessive backup items. A scheduled full backup is described by a custom BackupSchedule
CR object. A full backup is triggered at each backup time point. Its underlying implementation is the ad-hoc full backup.
Step 1. Prepare for a scheduled full backup
The steps to prepare for a scheduled full backup are the same as those of Prepare for an ad-hoc backup.
Step 2. Perform a scheduled full backup
Depending on which method you choose to grant permissions to the remote storage, perform a scheduled full backup by doing one of the following:
Method 1: If you grant permissions by access key, create the
BackupSchedule
CR, and back up cluster data as described below:kubectl apply -f backup-scheduler-azblob.yamlThe content of
backup-scheduler-azblob.yaml
is as follows:--- apiVersion: pingcap.com/v1alpha1 kind: BackupSchedule metadata: name: demo1-backup-schedule-azblob namespace: test1 spec: #maxBackups: 5 #pause: true maxReservedTime: "3h" schedule: "*/2 * * * *" backupTemplate: backupType: full br: cluster: demo1 clusterNamespace: test1 # logLevel: info # statusAddr: ${status_addr} # concurrency: 4 # rateLimit: 0 # timeAgo: ${time} # checksum: true # sendCredToTikv: true # Only needed for TiDB Operator < v1.1.10 or TiDB < v4.0.8 from: host: ${tidb_host} port: ${tidb_port} user: ${tidb_user} secretName: backup-demo1-tidb-secret azblob: secretName: azblob-secret-ad container: my-container prefix: my-folderMethod 2: If you grant permissions by Azure AD, create the
BackupSchedule
CR, and back up cluster data as described below:kubectl apply -f backup-scheduler-azblob.yamlThe content of
backup-scheduler-azblob.yaml
is as follows:--- apiVersion: pingcap.com/v1alpha1 kind: BackupSchedule metadata: name: demo1-backup-schedule-azblob namespace: test1 spec: #maxBackups: 5 #pause: true maxReservedTime: "3h" schedule: "*/2 * * * *" backupTemplate: backupType: full br: cluster: demo1 sendCredToTikv: false clusterNamespace: test1 # logLevel: info # statusAddr: ${status_addr} # concurrency: 4 # rateLimit: 0 # timeAgo: ${time} # checksum: true # Only needed for TiDB Operator < v1.1.10 or TiDB < v4.0.8 from: host: ${tidb_host} port: ${tidb_port} user: ${tidb_user} secretName: backup-demo1-tidb-secret azblob: secretName: azblob-secret-ad container: my-container prefix: my-folder
From the preceding content in backup-scheduler-azblob.yaml
, you can see that the backupSchedule
configuration consists of two parts. One is the unique configuration of backupSchedule
, and the other is backupTemplate
.
- For the unique configuration of
backupSchedule
, refer to BackupSchedule CR fields. backupTemplate
specifies the configuration related to the cluster and remote storage, which is the same as thespec
configuration of theBackup
CR.
After creating the scheduled full backup, you can run the following command to check the backup status:
kubectl get bks -n test1 -o wide
You can run the following command to check all the backup items:
kubectl get bk -l tidb.pingcap.com/backup-schedule=demo1-backup-schedule-azblob -n test1
Delete the backup CR
If you no longer need the backup CR, you can delete it by referring to Delete the Backup CR.
Troubleshooting
If you encounter any problem during the backup process, refer to Common Deployment Failures.