TiCDC Changefeeds の CLI およびコンフィグレーションパラメーター

Changefeed CLI パラメータ

このセクションでは、レプリケーション (changefeed) タスクを作成する方法を説明することで、TiCDC changefeeds のコマンド ライン パラメーターを紹介します。

cdc cli changefeed create --server= --sink-uri="mysql://root:123456@" --changefeed-id="simple-replication-task"
Create changefeed successfully! ID: simple-replication-task Info: {"upstream_id":7178706266519722477,"namespace":"default","id":"simple-replication-task","sink_uri":"mysql://root:xxxxx@","create_time":"2023-03-10T15:05:46.679218+08:00","start_ts":438156275634929669,"engine":"unified","config":{"case_sensitive":true,"enable_old_value":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"bdr_mode":false,"sync_point_interval":30000000000,"sync_point_retention":3600000000000,"filter":{"rules":["test.*"],"event_filters":null},"mounter":{"worker_num":16},"sink":{"protocol":"","schema_registry":"","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false},"column_selectors":null,"transaction_atomicity":"none","encoder_concurrency":16,"terminator":"\r\n","date_separator":"none","enable_partition_separator":false},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"storage":""}},"state":"normal","creator_version":"v6.5.2"}
  • --changefeed-id : レプリケーション タスクの ID。形式は^[a-zA-Z0-9]+(\-[a-zA-Z0-9]+)*$正規表現と一致する必要があります。この ID が指定されていない場合、TiCDC は ID として UUID (バージョン 4 形式) を自動的に生成します。

  • --sink-uri : レプリケーション タスクのダウンストリーム アドレス。 --sink-uri次の形式に従って構成します。現在、スキームはmysqltidb 、およびkafkaをサポートしています。


    シンク URI に! * ' ( ) ; : @ & = + $ , / ? % # [ ]などの特殊文字が含まれている場合は、特殊文字をエスケープする必要があります (たとえば、 URI エンコーダーなど)。

  • --start-ts : 変更フィードの開始 TSO を指定します。この TSO から、TiCDC クラスターはデータのプルを開始します。デフォルト値は現在の時刻です。

  • --target-ts : changefeed の終了 TSO を指定します。この TSO に対して、TiCDC クラスターはデータのプルを停止します。デフォルト値は空です。これは、TiCDC がデータのプルを自動的に停止しないことを意味します。

  • --config : changefeed の構成ファイルを指定します。

Changefeed 構成パラメーター

このセクションでは、レプリケーション タスクの構成について説明します。

# Specifies whether the database names and tables in the configuration file are case-sensitive. # The default value is true. # This configuration item affects configurations related to filter and sink. case-sensitive = true # Specifies whether to output the old value. New in v4.0.5. Since v5.0, the default value is `true`. enable-old-value = true # Specifies whether to enable the Syncpoint feature, which is supported since v6.3.0 and is disabled by default. # Since v6.4.0, only the changefeed with the SYSTEM_VARIABLES_ADMIN or SUPER privilege can use the TiCDC Syncpoint feature. # enable-sync-point = false # Specifies the interval at which Syncpoint aligns the upstream and downstream snapshots. # The format is in h m s. For example, "1h30m30s". # The default value is "10m" and the minimum value is "30s". # sync-point-interval = "5m" # Specifies how long the data is retained by Syncpoint in the downstream table. When this duration is exceeded, the data is cleaned up. # The format is in h m s. For example, "24h30m30s". # The default value is "24h". # sync-point-retention = "1h" [mounter] # The number of threads with which the mounter decodes KV data. The default value is 16. # worker-num = 16 [filter] # Ignores the transaction of specified start_ts. # ignore-txn-start-ts = [1, 2] # Filter rules. # Filter syntax: <https://docs.pingcap.com/tidb/stable/table-filter#syntax>. rules = ['*.*', '!test.*'] # Event filter rules. # The detailed syntax is described in the event filter rules section. # The first event filter rule. [[filter.event-filters]] matcher = ["test.worker"] # matcher is an allow list, which means this rule only applies to the worker table in the test database. ignore-event = ["insert"] # Ignore insert events. ignore-sql = ["^drop", "add column"] # Ignore DDLs that start with "drop" or contain "add column". ignore-delete-value-expr = "name = 'john'" # Ignore delete DMLs that contain the condition "name = 'john'". ignore-insert-value-expr = "id >= 100" # Ignore insert DMLs that contain the condition "id >= 100". ignore-update-old-value-expr = "age < 18" # Ignore update DMLs whose old value contains "age < 18". ignore-update-new-value-expr = "gender = 'male'" # Ignore update DMLs whose new value contains "gender = 'male'". # The second event filter rule. matcher = ["test.fruit"] # matcher is an allow list, which means this rule only applies to the fruit table in the test database. ignore-event = ["drop table"] # Ignore drop table events. ignore-sql = ["delete"] # Ignore delete DMLs. ignore-insert-value-expr = "price > 1000 and origin = 'no where'" # Ignore insert DMLs that contain the conditions "price > 1000" and "origin = 'no where'". [sink] # For the sink of MQ type, you can use dispatchers to configure the event dispatcher. # Since v6.1.0, TiDB supports two types of event dispatchers: partition and topic. For more information, see <partition and topic link>. # The matching syntax of matcher is the same as the filter rule syntax. For details about the matcher rules, see <>. dispatchers = [ {matcher = ['test1.*', 'test2.*'], topic = "Topic expression 1", partition = "ts" }, {matcher = ['test3.*', 'test4.*'], topic = "Topic expression 2", partition = "index-value" }, {matcher = ['test1.*', 'test5.*'], topic = "Topic expression 3", partition = "table"}, {matcher = ['test6.*'], partition = "ts"} ] # The protocol configuration item specifies the protocol format of the messages sent to the downstream. # When the downstream is Kafka, the protocol can only be canal-json or avro. # When the downstream is a storage service, the protocol can only be canal-json or csv. protocol = "canal-json" # The following three configuration items are only used when you replicate data to storage sinks and can be ignored when replicating data to MQ or MySQL sinks. # Row terminator, used for separating two data change events. The default value is an empty string, which means "\r\n" is used. terminator = '' # Date separator type used in the file directory. Value options are `none`, `year`, `month`, and `day`. `none` is the default value and means that the date is not separated. For more information, see <https://docs.pingcap.com/tidb/dev/ticdc-sink-to-cloud-storage#data-change-records>. date-separator = 'none' # Whether to use partitions as the separation string to store partitions in a table in separate directories. In v6.5.0 and v6.5.1, the default value is false. In v6.5.2 or a later v6.5.x version, the default value is true. It is recommended that you keep the value as `true` to avoid potential data loss in downstream partitioned tables <https://github.com/pingcap/tiflow/issues/8724>. For usage examples, see <https://docs.pingcap.com/tidb/dev/ticdc-sink-to-cloud-storage#data-change-records)>. enable-partition-separator = false # Since v6.5.0, TiCDC supports saving data changes to storage services in CSV format. Ignore the following configurations if you replicate data to MQ or MySQL sinks. [sink.csv] # The character used to separate fields in the CSV file. The value must be an ASCII character and defaults to `,`. delimiter = ',' # The quotation character used to surround fields in the CSV file. The default value is `"`. If the value is empty, no quotation is used. quote = '"' # The character displayed when a CSV column is null. The default value is `\N`. null = '\N' # Whether to include commit-ts in CSV rows. The default value is false. include-commit-ts = false


登録なしで TiDB の機能をワンストップでインタラクティブに体験できます。
TiDB Cloud
PoC お問い合わせ
OSS Insight
© 2023 PingCAP. All Rights Reserved.
Privacy Policy.