📣
TiDB Cloud Premium is now in public preview. Unlimited growth, instant elasticity, advanced security for enterprise workloads. Try it out →
​
Sign InStart for Free
AI
Developer
Best Practices
API
​
Sign InStart for Free

Integration Tasks

An integration task in TiDB Cloud Lake defines how data flows from a source into TiDB Cloud Lake. Each task references an existing data source and specifies source settings, a target location or result viewing method, and runtime parameters that are specific to the task type.

Unlike data sources, integration tasks are the executable units that actually perform data movement, synchronization, or message consumption. Data sources store access settings, while tasks handle scheduling, ingestion, synchronization, consumption, stopping, resuming, and monitoring.

Supported Task Types

Task TypeDescription
Amazon S3Imports CSV, Parquet, or NDJSON files from Amazon S3 with support for one-time or continuous ingestion.
Amazon SQS (S3) (Beta)Consumes S3 object creation events from an SQS queue and writes the corresponding object data into TiDB Cloud Lake.
MySQLSynchronizes table data from MySQL using Snapshot, CDC Only, or Snapshot + CDC.
PostgreSQLSynchronizes table data from PostgreSQL using Snapshot, CDC Only, or Snapshot + CDC.
Kafka Consumer Integration Task (Beta)Continuously consumes messages from Kafka topics and saves the message content to internal object storage.

Reading Guide

Recommended reading order:

  1. Start with Task Management to understand the task creation flow, start / stop behavior, status, and run history.
  2. Then read the task-specific guide for the source type you want to configure.

Task Type Differences

  • S3 tasks are designed for file import scenarios and mainly focus on file path patterns, file formats, and ingestion behavior.
  • SQS (S3) tasks are designed for S3 event-driven data ingestion and mainly focus on the SQS queue, S3 event filters, IAM Role, and target table.
  • MySQL and PostgreSQL tasks are designed for table synchronization scenarios and mainly focus on sync modes, primary keys, incremental capture, and archive scheduling.
  • Kafka Consumer tasks are designed for message consumption scenarios and mainly focus on topics, start position, batch size, batch wait interval, and tenant Stage queries.

Was this page helpful?