📣
TiDB Cloud Premium is now in public preview. Unlimited growth, instant elasticity, advanced security for enterprise workloads. Try it out →
​
Sign InStart for Free
AI
Developer
Best Practices
API
​
Sign InStart for Free

Data Integration Overview

The Data Integration feature in TiDB Cloud Lake provides a visual, no-code interface for importing, synchronizing, or consuming data from external systems into TiDB Cloud Lake. The feature centers around two key concepts: data sources and integration tasks.

Key Concepts

ConceptDescription
Data SourcesReusable connection settings or credentials used to access external systems or send notifications, such as AWS Access Key / Secret Key, MySQL hostname / username / password, SQS (S3) queue URL, Kafka broker addresses, or a FeiShu bot webhook.
Integration TasksExecutable tasks that define where data comes from, where the task writes data or how it saves results, which runtime parameters it uses, and how you start and monitor the task.

Data sources do not move data by themselves. They only store the information required to access external systems. Integration tasks are the units that actually perform imports, snapshots, continuous synchronization, or message consumption.

Not every data source corresponds to an ingestion task. For example, FeiShuBot is used for notifications rather than loading source data into TiDB Cloud Lake.

Supported Integration Task Types

Task TypeDescription
Amazon S3Imports CSV, Parquet, or NDJSON files from Amazon S3 with support for one-time or continuous ingestion.
Amazon SQS (S3) (Beta)Consumes S3 object creation events from an SQS queue and writes the corresponding object data into TiDB Cloud Lake.
MySQLSynchronizes table data from MySQL using Snapshot, CDC Only, or Snapshot + CDC modes.
PostgreSQLSynchronizes table data from PostgreSQL using Snapshot, CDC Only, or Snapshot + CDC modes.
Kafka Consumer Integration Task (Beta)Continuously consumes messages from Kafka topics and saves the message content to internal object storage.
  1. Create and test reusable connection settings on the Data Sources page.
  2. Review supported task types and their use cases on the Integration Tasks page.
  3. Read the task-specific guide to configure the source, preview the data, and configure the result location or result viewing method.
  4. Use the Task Management page to start tasks, check status, and troubleshoot execution issues.

Was this page helpful?