- About TiDB Cloud
- Get Started
- Manage Cluster
- Plan Your Cluster
- Create a TiDB Cluster
- Connect to Your TiDB Cluster
- Set Up VPC Peering Connections
- Use an HTAP Cluster
- Scale a TiDB Cluster
- Tune Performance
- Upgrade a TiDB Cluster
- Delete a TiDB Cluster
- Manage Data
- Import Sample Data
- Migrate Data into TiDB
- Back Up and Restore Data
- Export Data from TiDB
- Monitor Cluster
- Overview
- Built-in Alerting
- Third-Party Monitoring Integrations
- Manage User Access
- Manage Billing
- Reference
- Release Notes
Sink to Apache Kafka
Currently, Sink to Apache Kafka is an experimental feature. It is not recommended that you use it for production environments.
This document describes how to stream data from TiDB Cloud to Apache Kafka using the Sink to Apache Kafka changefeed.
Prerequisites
Network
Make sure that your TiDB Cluster can connect to the Apache Kafka service.
If your Apache Kafka service is in an AWS VPC that has no internet access, take the following steps:
Set up a VPC peering connection between the VPC of the Apache Kafka service and your TiDB cluster.
Modify the inbound rules of the security group that the Apache Kafka service is associated with.
You must add the CIDR of the region where your TiDB Cloud cluster is located to the inbound rules. The CIDR can be found on the VPC Peering page. Doing so allows the traffic to flow from your TiDB cluster to the Kafka brokers.
If the Apache Kafka URL contains hostnames, you need to allow TiDB Cloud to be able to resolve the DNS hostnames of the Apache Kafka brokers.
- Follow the steps in Enable DNS resolution for a VPC peering connection.
- Enable the Accepter DNS resolution option.
If your Apache Kafka service is in a GCP VPC that has no internet access, take the following steps:
Set up a VPC peering connection between the VPC of the Apache Kafka service and your TiDB cluster.
Modify the ingress firewall rules of the VPC where Apache Kafka is located.
You must add the CIDR of the Region where your TiDB Cloud cluster is located to the ingress firewall rules. The CIDR can be found on the VPC Peering page. Doing so allows the traffic to flow from your TiDB cluster to the Kafka brokers.
Topic
You must prepare a Topic before creating an Apache Kafka Sink. Based on table, the Sink will distribute data to different partitions of the Topic.
Create a Sink
After completing the Prerequisites, you can sink your data to Apache Kafka.
- Navigate to the Changefeed tab of your TiDB cluster.
- Click Sink to Apache Kafka.
- Fill the Kafka URL and Kafka Topic.
- Click Test Connectivity. If your TiDB Cluster can connect to the Apache Kafka service, the Confirm button is displayed.
- Click Confirm and after a while, the sink will begin its work, and the status of the sink will be changed to "Producing".
Delete a Sink
- Navigate to the Changefeed tab of a cluster.
- Click the trash button of Sink to Apache Kafka
Restrictions
Because TiDB Cloud uses TiCDC to establish connectors, it has the same restrictions as TiCDC.