URI Formats of External Storage Services

This document describes the URI formats of external storage services, including Amazon S3, GCS, and Azure Blob Storage.

The basic format of the URI is as follows:

[scheme]://[host]/[path]?[parameters]

Amazon S3 URI format

• scheme: s3

• host: bucket name

• parameters:

  • access-key: Specifies the access key.

  • secret-access-key: Specifies the secret access key.

  • session-token: Specifies the temporary session token.

  • use-accelerate-endpoint: Specifies whether to use the accelerate endpoint on Amazon S3 (defaults to false).

  • endpoint: Specifies the URL of custom endpoint for S3-compatible services (for example, <https://s3.example.com/>).

  • force-path-style: Use path style access rather than virtual hosted style access (defaults to true).

  • storage-class: Specifies the storage class of the uploaded objects (for example, STANDARD or STANDARD_IA).

  • sse: Specifies the server-side encryption algorithm used to encrypt the uploaded objects (value options: empty, AES256, or aws:kms).

  • sse-kms-key-id: Specifies the KMS ID if sse is set to aws:kms.

  • acl: Specifies the canned ACL of the uploaded objects (for example, private or authenticated-read).

  • role-arn: To allow TiDB Cloud to access Amazon S3 data using a specific IAM role, provide the role's Amazon Resource Name (ARN) in the role-arn URL query parameter. For example: arn:aws:iam::888888888888:role/my-role.

    Note

    • To automatically create an IAM role, navigate to the Import Data from Amazon S3 page of your cluster in the TiDB Cloud console, fill in the Folder URI field, click Click here to create new one with AWS CloudFormation under the Role ARN field, and then follow the on-screen instructions in the Add New Role ARN dialog.
    • If you have any trouble creating the IAM role using AWS CloudFormation, click click Having trouble? Create Role ARN manually in the Add New Role ARN dialog to get the TiDB Cloud Account ID and TiDB Cloud External ID, and then follow the steps in Configure Amazon S3 access using a Role ARN to create the role manually. When configuring the IAM role, make sure to enter the TiDB Cloud account ID in the Account ID field and select Require external ID to protect against confused deputy attacks.
    • To enhance security, you can reduce the valid duration of the IAM role by configuring a shorter Max session duration. For more information, see Update the maximum session duration for a role in AWS documentation.

  • external-id: Specifies the TiDB Cloud External ID, which is required for TiDB Cloud to access Amazon S3 data. You can obtain this ID from the Add New Role ARN dialog in the TiDB Cloud console. For more information, see Configure Amazon S3 access using a Role ARN.

The following is an example of an Amazon S3 URI for BACKUP and RESTORE. This example uses the file path testfolder.

s3://external/testfolder?access-key=${access-key}&secret-access-key=${secret-access-key}

GCS URI format

• scheme: gcs or gs

• host: bucket name

• parameters:

  • credentials-file: Specifies the path to the credentials JSON file on the migration tool node.
  • storage-class: Specifies the storage class of the uploaded objects (for example, STANDARD or COLDLINE)
  • predefined-acl: Specifies the predefined ACL of the uploaded objects (for example, private or project-private)

The following is an example of a GCS URI for IMPORT INTO. In this example, you need to specify a specific filename test.csv.

gcs://external/test.csv?credentials-file=${credentials-file-path}

Azure Blob Storage URI format

• scheme: azure or azblob

• host: container name

• parameters:

  • account-name: Specifies the account name of the storage.
  • account-key: Specifies the access key.
  • sas-token: Specifies the shared access signature (SAS) token.
  • access-tier: Specifies the access tier of the uploaded objects, for example, Hot, Cool, or Archive. The default value is the default access tier of the storage account.
  • encryption-scope: Specifies the encryption scope for server-side encryption.
  • encryption-key: Specifies the encryption key for server-side encryption, which uses the AES256 encryption algorithm.

The following is an example of an Azure Blob Storage URI for BR. In this example, you need to specify a specific file path testfolder.

azure://external/testfolder?account-name=${account-name}&account-key=${account-key}