Grant Permissions to Remote Storage

This document describes how to grant permissions to access remote storage for backup and restore. During the backup process, TiDB cluster data is backed up to the remote storage. During the restore process, the backup data is restored from the remote storage to the TiDB cluster.

AWS account permissions

Amazon Web Service (AWS) provides different methods to grant permissions for different types of Kubernetes clusters. This document describes the following three methods.

Grant permissions by AccessKey and SecretKey

The AWS client can read AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the process environment variables to obtain the associated user or role permissions.

Create the s3-secret secret by running the following command. Use the AWS account's AccessKey and SecretKey. The secret stores the credential used for accessing S3-compatible storage.

kubectl create secret generic s3-secret --from-literal=access_key=xxx --from-literal=secret_key=yyy --namespace=test1

Grant permissions by associating IAM with Pod

If you associate the user's IAM role with the resources of the running Pods, the processes running in the Pods can have the permissions of the role. This method is provided by kube2iam.

1. Create an IAM role.

   First, create an IAM User for your account.

   Then, Give the required permission to the IAM role you have created. Refer to Adding and Removing IAM Identity Permissions for details.

   Because the Backup CR needs to access the Amazon S3 storage, the IAM role is granted the AmazonS3FullAccess permission.

   When backing up a TiDB cluster using EBS volume snapshots, besides the AmazonS3FullAccess permission, the following permissions are also required:

           {
           "Effect": "Allow",
           "Action": [
               "ec2:AttachVolume",
               "ec2:CreateSnapshot",
               "ec2:CreateSnapshots",
               "ec2:CreateTags",
               "ec2:CreateVolume",
               "ec2:DeleteSnapshot",
               "ec2:DeleteTags",
               "ec2:DeleteVolume",
               "ec2:DescribeInstances",
               "ec2:DescribeSnapshots",
               "ec2:DescribeTags",
               "ec2:DescribeVolumes",
               "ec2:DetachVolume",
               "ebs:ListSnapshotBlocks",
               "ebs:ListChangedBlocks"
           ],
           "Resource": "*"
       }
   

2. Associate IAM with the TiKV Pod:

   When you use BR to back up TiDB data, the TiKV Pod also needs to perform read and write operations on S3-compatible storage as the BR Pod does. Therefore, you need to add annotations to the TiKV Pod to associate it with the IAM role.

   kubectl patch tc demo1 -n test1 --type merge -p '{"spec":{"tikv":{"annotations":{"iam.amazonaws.com/role":"arn:aws:iam::123456789012:role/user"}}}}'
   

   After the TiKV Pod is restarted, check whether the Pod has the annotation.

Grant permissions by associating IAM with ServiceAccount

If you associate the user's IAM role with serviceAccount of Kubernetes, the Pods using the serviceAccount can have the permissions of the role. This method is provided by EKS Pod Identity Webhook.

When you use this method to grant permissions, you can create the EKS cluster and deploy TiDB Operator and the TiDB cluster.

1. Enable the IAM role for the serviceAccount in the cluster:

   Refer to AWS documentation.

2. Create the IAM role:

   Create an IAM role and grant the AmazonS3FullAccess permissions to the role. Edit the role's Trust relationships to grant tidb-backup-manager the access to this IAM role.

   When backing up a TiDB cluster using EBS volume snapshots, besides the AmazonS3FullAccess permission, the following permissions are also required:

           {
           "Effect": "Allow",
           "Action": [
               "ec2:AttachVolume",
               "ec2:CreateSnapshot",
               "ec2:CreateSnapshots",
               "ec2:CreateTags",
               "ec2:CreateVolume",
               "ec2:DeleteSnapshot",
               "ec2:DeleteTags",
               "ec2:DeleteVolume",
               "ec2:DescribeInstances",
               "ec2:DescribeSnapshots",
               "ec2:DescribeTags",
               "ec2:DescribeVolumes",
               "ec2:DetachVolume",
               "ebs:ListSnapshotBlocks",
               "ebs:ListChangedBlocks"
           ],
           "Resource": "*"
       }
   

   At the same time, edit the role's Trust relationships to grant tidb-controller-manager the access to this IAM role.

3. Associate the IAM role with the ServiceAccount resources.

   kubectl annotate sa tidb-backup-manager eks.amazonaws.com/role-arn=arn:aws:iam::123456789012:role/user --namespace=test1
   

   When backing up or restoring a TiDB cluster using EBS volume snapshots, you need to associate the IAM role with the ServiceAccount resources of tidb-controller-manager.

   kubectl annotate sa tidb-controller-manager eks.amazonaws.com/role-arn=arn:aws:iam::123456789012:role/user --namespace=tidb-admin
   

   Restart the tidb-controller-manager Pod of TiDB Operator to make the configured ServiceAccount take effect.

4. Associate the ServiceAccount with the TiKV Pod:

   kubectl patch tc demo1 -n test1 --type merge -p '{"spec":{"tikv":{"serviceAccount": "tidb-backup-manager"}}}'
   

   Modify the value of spec.tikv.serviceAccount to tidb-backup-manager. After the TiKV Pod is restarted, check whether the Pod's serviceAccountName is changed.

5. (Optional) If your cluster includes TiFlash Pods, repeat step 4 to associate the ServiceAccount with the TiFlash Pod.

   kubectl patch tc demo1 -n test1 --type merge -p '{"spec":{"tiflash":{"serviceAccount": "tidb-backup-manager"}}}'
   

GCS account permissions

Grant permissions by the service account

Create the gcs-secret secret which stores the credential used to access GCS. The google-credentials.json file stores the service account key that you have downloaded from the Google Cloud console. Refer to Google Cloud documentation for details.

kubectl create secret generic gcs-secret --from-file=credentials=./google-credentials.json -n test1

Azure account permissions

Azure provides different methods to grant permissions for different types of Kubernetes clusters. This document describes the following two methods.

Grant permissions by access key

The Azure client can read AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY from the process environment variables to obtain the associated user or role permissions.

Run the following command to create the azblob-secret secret and use your Azure account access key to grant permissions. The secret stores the credential used for accessing Azure Blob Storage.

kubectl create secret generic azblob-secret --from-literal=AZURE_STORAGE_ACCOUNT=xxx --from-literal=AZURE_STORAGE_KEY=yyy --namespace=test1

Grant permissions by Azure AD

The Azure client can read AZURE_STORAGE_ACCOUNT, AZURE_CLIENT_ID, AZURE_TENANT_ID, and AZURE_CLIENT_SECRET to obtain the associated user or role permissions.

1. Create the azblob-secret-ad secret by running the following command. Use the Active Directory (AD) of your Azure account. The secret stores the credential used for accessing Azure Blob Storage.

   kubectl create secret generic azblob-secret-ad --from-literal=AZURE_STORAGE_ACCOUNT=xxx --from-literal=AZURE_CLIENT_ID=yyy --from-literal=AZURE_TENANT_ID=zzz --from-literal=AZURE_CLIENT_SECRET=aaa --namespace=test1
   

2. Associate the secret with the TiKV Pod:

   When you use BR to back up TiDB data, the TiKV Pod also needs to perform read and write operations on Azure Blob Storage as the BR Pod does. Therefore, you need to associate the TiKV Pod with the secret.

   kubectl patch tc demo1 -n test1 --type merge -p '{"spec":{"tikv":{"envFrom":[{"secretRef":{"name":"azblob-secret-ad"}}]}}}'
   

   After the TiKV Pod is restarted, check whether the Pod has the environment variables.