📣

TiDB Cloud Premium is now in public preview. Unlimited growth, instant elasticity, advanced security for enterprise workloads. Try it out →

Set Up an External Stage for TiDB Cloud Data Pipeline (AWS)

This guide explains how to prepare an Amazon S3 bucket as the external stage for the TiDB Cloud Data Pipeline. An external stage is the intermediate bucket where TiDB Cloud writes exported snapshots and row changes, and TiDB Cloud Lake reads from it to load data into the target warehouse.

TiDB Cloud writes data to your S3 bucket, and TiDB Cloud Lake reads data from it.

Prerequisites

An AWS account with permissions to manage IAM, S3, and optionally SQS resources.
A TiDB Cloud account with a TiDB Cloud Lake warehouse.
An S3 bucket in the same region as your TiDB Cloud instance. If you do not have one yet, create it in Create an S3 Bucket.

Step 1. Create an S3 bucket

Tip

If you already have an S3 bucket ready, skip this step and just make sure your bucket region matches the region of your TiDB Cloud instance.

Open the AWS S3 Console and create a new bucket.
Select a region, and make sure this region matches the region of your TiDB Cloud instance.
(Optional) Create a folder (prefix) inside the bucket to organize TiDB Cloud data (for example, s3://tidb-cloud-lake-data/my-cluster/).

Step 2. Configure bucket access

Choose one of the following options for bucket access, and then complete the steps in the corresponding section:

Option 1: Bucket access with role ARN (CloudFormation) (recommended)
Option 2: Bucket access with role ARN (manual setup)
Option 3: Bucket access with access key (not recommended)

Tip

Decide whether to enable event-driven ingestion before you start.

By default, TiDB Cloud Lake periodically scans the bucket for new data after the data pipeline is created. If you need lower data latency, you can optionally enable event-driven ingestion with an SQS queue. When new data is written to the bucket, an S3 event notification is sent to the SQS queue, allowing TiDB Cloud Lake to detect and load the new data without waiting for the next scheduled scan. The changefeed still flushes data to the bucket according to its configured cadence. Because event-driven ingestion can cause TiDB Cloud Lake to load data more frequently, it can increase the TiDB Cloud Lake service hosting cost.

With option 1, if you choose to enable event-driven ingestion, you can have the CloudFormation stack create the SQS queue when you create the stack, or add an SQS queue later manually.
With options 2 and 3, if you choose to enable event-driven ingestion, you need to create and configure the SQS queue manually.

Option 1. Bucket access with role ARN (CloudFormation)

One IAM role is shared by TiDB Cloud (which writes to the bucket) and TiDB Cloud Lake (which reads from it). The role's trust policy allows both sides to assume it, each guarded by its own external ID, so no long-lived credential is stored. This is the recommended option because the CloudFormation stack creates the role, its trust relationship, its permissions, and optionally the SQS queue and its policy in one go.

1.1 Create the role with CloudFormation

In the TiDB Cloud console, open the Create Data Pipeline page, go to the External Stage area, and enter your Bucket URI.
Under Bucket Access, select AWS Role ARN, and click the CloudFormation link below the field to open the dialog.
Click AWS Console with CloudFormation Template. A new browser tab opens the AWS CloudFormation console with all parameters pre-filled.
In the new tab, enter a stack name, and optionally enter an SQS queue name if you need event-driven ingestion. CloudFormation creates the queue and its notification policy automatically. Leave the SQS field empty to skip it.
Create the stack and wait until the status becomes CREATE_COMPLETE.
In the Outputs tab of the stack, record the values required when you configure the External Stage in the TiDB Cloud console:
- Role ARN: the RoleARN value.
- SQS queue URL (only if you enabled SQS): the queue URL in the format https://sqs.<region>.amazonaws.com/<account-id>/<queue-name>.

1.2 (Optional) Configure the S3 bucket notification

If you enabled SQS in 1.1, the queue and its policy are already created by the stack. The only remaining step is to configure the notification on your bucket, which the provided CloudFormation stack does not configure because the bucket already exists. Follow the manual notification steps in 2.3.2.

Option 2. Bucket access with role ARN (manual setup)

Use this option if you cannot use CloudFormation, or if your organization requires every IAM resource to be created and reviewed manually. The IAM role itself is the same as in option 1; only the way you create it is different.

2.1 Collect the required values

In the TiDB Cloud console, open the Create Data Pipeline page, go to the External Stage area, and enter your Bucket URI.
Under Bucket Access, select AWS Role ARN, and click the CloudFormation link below the field to open the dialog.
Copy the following values from the Having trouble? area of the dialog. You need all of them for the trust policy in 2.2:
- TiDB Cloud account ID
- TiDB Cloud external ID
- Lake external ID
- Lake platform setup & validation role ARN
- Lake platform data loading role ARN

2.2 Create the role and attach the policies

Open the IAM Console, go to Roles > Create role.

Under Trusted entity type, select Custom trust policy, and paste the following into the policy document. Replace the placeholder values with the ones you collected:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowTiDBCloudAssumeRole",
      "Effect": "Allow",
      "Principal": { "AWS": "<TiDB Cloud account ID>" },
      "Action": "sts:AssumeRole",
      "Condition": { "StringEquals": { "sts:ExternalId": "<TiDB Cloud external ID>" } }
    },
    {
      "Sid": "AllowLakeSetupAssumeRole",
      "Effect": "Allow",
      "Principal": { "AWS": "<Lake platform setup & validation role ARN>" },
      "Action": "sts:AssumeRole",
      "Condition": { "StringEquals": { "sts:ExternalId": "<Lake external ID>" } }
    },
    {
      "Sid": "AllowLakeLoadAssumeRole",
      "Effect": "Allow",
      "Principal": { "AWS": "<Lake platform data loading role ARN>" },
      "Action": "sts:AssumeRole",
      "Condition": { "StringEquals": { "sts:ExternalId": "<Lake external ID>" } }
    }
  ]
}

Enter a role name (for example, tidb-cloud-lake-role) and click Create role.
Open the role you just created, go to the Permissions tab, and click Add permissions > Create inline policy.

Select the JSON tab, and paste the following. Replace YOUR_BUCKET_NAME and your-prefix with your actual values, and remove the SQSConsumeAccess statement if you do not need event-driven ingestion:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3BucketMetadata",
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
    },
    {
      "Sid": "S3ObjectReadWrite",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/your-prefix/*"
    },
    {
      "Sid": "SQSConsumeAccess",
      "Effect": "Allow",
      "Action": ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes", "sqs:ChangeMessageVisibility"],
      "Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:YOUR_QUEUE_NAME"
    }
  ]
}

Enter a policy name (for example, tidb-cloud-lake-access) and click Create policy.
Copy the role ARN from the role details page. You need it when you configure the External Stage in the TiDB Cloud console, for example arn:aws:iam::123456789012:role/tidb-cloud-lake-role.

2.3 (Optional) Enable event-driven ingestion with SQS

Skip this section if periodic scanning is acceptable for your workload. For details about when to use SQS, see the Tip in Step 2. Configure bucket access.

2.3.1 Create the SQS queue and configure the queue policy

Open the SQS Console, click Create queue, select Standard type, and enter a name (for example, tidb-cloud-lake-sqs). Click Create queue.

Open the queue, go to the Access policy tab, and replace the policy with the following. This allows S3 to send notifications to the queue:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowS3ToSendMessage",
      "Effect": "Allow",
      "Principal": { "Service": "s3.amazonaws.com" },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:YOUR_QUEUE_NAME",
      "Condition": {
        "ArnLike": { "aws:SourceArn": "arn:aws:s3:::YOUR_BUCKET_NAME" },
        "StringEquals": { "aws:SourceAccount": "ACCOUNT_ID" }
      }
    }
  ]
}

Note

Make sure the SQSConsumeAccess statement from 2.2 is included in the role's permissions policy.
Record the queue URL from the queue details page in the SQS console. You need it when you configure the External Stage in the TiDB Cloud console, in the format https://sqs.<region>.amazonaws.com/<account-id>/<queue-name>.

2.3.2 Configure the S3 bucket notification

Configure an S3 event notification to send object creation events from your bucket to the SQS queue:

Open the AWS S3 Console and navigate to your bucket.
Go to Properties > Event notifications > Create event notification.
Configure:
- Event types: select All object create events.
- Destination: select SQS queue and choose the queue created earlier.
Click Save changes.

Option 3. Bucket access with access key (not recommended)

Note

Using Access Key/Secret Key (AK/SK) means you need to manage and rotate credentials manually, and there is a higher risk of accidental exposure. For simpler management and better security, it is recommended to follow the steps in Option 1 or Option 2 to create a Role ARN instead.

With this option, you create an IAM user and provide its Access Key ID and Secret Access Key to TiDB Cloud. TiDB Cloud accesses your S3 bucket directly with these credentials.

3.1 Create an IAM user and access key

Open the IAM Console and go to Users > Create user.
Enter a user name (for example, tidb-cloud-lake-user), and click Next.

On the Set permissions page, create or attach a policy that grants the required S3 and optional SQS permissions. Replace YOUR_BUCKET_NAME and your-prefix with your actual values, and remove the SQSConsumerAccess statement if you do not need event-driven ingestion:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3BucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR_BUCKET_NAME",
        "arn:aws:s3:::YOUR_BUCKET_NAME/your-prefix/*"
      ]
    },
    {
      "Sid": "SQSConsumerAccess",
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "sqs:ChangeMessageVisibility"
      ],
      "Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:YOUR_QUEUE_NAME"
    }
  ]
}

Click Next. On the Review and create page, review the user settings, and then click Create user.
On the Users page, click the name of the user you just created, and go to the Security credentials tab.
In the Access keys section, click Create access key. On the Access key best practices & alternatives page, select Other, click Next, and then create the access key.
Save the Access Key ID and Secret Access Key. You need them when configuring the External Stage in the TiDB Cloud console.
Note
The Secret Access Key is only shown once at creation time. Make sure to save it immediately.

3.2 (Optional) Enable event-driven ingestion with SQS

Skip this section if periodic scanning is acceptable for your workload.

Open the SQS Console, click Create queue, select Standard type, and enter a name (for example, tidb-cloud-lake-sqs). Click Create queue.

Open the queue, go to the Access policy tab, and replace the policy with the following, so that S3 can send notifications to the queue. Replace the placeholder values with your actual values:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowS3ToSendMessage",
      "Effect": "Allow",
      "Principal": { "Service": "s3.amazonaws.com" },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:YOUR_QUEUE_NAME",
      "Condition": {
        "ArnLike": { "aws:SourceArn": "arn:aws:s3:::YOUR_BUCKET_NAME" },
        "StringEquals": { "aws:SourceAccount": "ACCOUNT_ID" }
      }
    }
  ]
}

Configure the notification on your bucket:
1. Open the AWS S3 Console and navigate to your bucket.
2. Go to Properties > Event notifications > Create event notification.
3. Configure:
  - Event types: select All object create events.
  - Destination: select SQS queue and choose the queue created earlier.
4. Click Save changes.
Record the queue URL from the queue details page in the SQS console. You need it when you configure the External Stage in the TiDB Cloud console, in the format https://sqs.<region>.amazonaws.com/<account-id>/<queue-name>.

What's next?

After completing the AWS setup, you have all the values required by the External Stage configuration:

S3 URI: from Create an S3 bucket.
Bucket access: the Role ARN (options 1 and 2), or the Access Key ID and Secret Access Key (option 3).
SQS queue URL (optional).

In the TiDB Cloud console, navigate to the Data Pipeline configuration page for your TiDB Cloud instance, and enter these values in the External Stage settings to complete the data pipeline setup.