TiDB 6.5.4 Release Notes

Release date: August 28, 2023

TiDB version: 6.5.4

Quick access: Quick start | Production deployment

Compatibility changes

• To fix the issue that TiDB consumes too much memory when using Cursor Fetch to fetch a large result set, TiDB automatically writes the result set to the disk to release memory #43233 @YangKeao
• Disable periodic compaction of RocksDB by default, so that the default behavior of TiKV RocksDB is now consistent with that in versions before v6.5.0. This change prevents potential performance impact caused by a significant number of compactions after upgrading. In addition, TiKV introduces two new configuration items rocksdb.[defaultcf|writecf|lockcf].periodic-compaction-seconds and rocksdb.[defaultcf|writecf|lockcf].ttl, enabling you to manually configure periodic compaction of RocksDB #15355 @LykxSassinator

Behavior changes

• For transactions containing multiple changes, if the primary key or non-null unique index value is modified in the update event, TiCDC splits an event into delete and insert events and ensures that all events follow the sequence of delete events preceding insert events. For more information, see documentation.

Improvements

• TiDB

  • Optimize the performance of LOAD DATA statements that contain assignment expressions #46081 @gengliqi
  • Optimize the performance of reading the dumped chunks from disk #45125 @YangKeao
  • Add a halt-scheduling configuration item to suspend PD scheduling #6493 @JmPotato

• TiKV

  • Use gzip compression for check_leader requests to reduce traffic #14553 @you06
  • Add the Max gap of safe-ts and Min safe ts region metrics and introduce the tikv-ctl get-region-read-progress command to better observe and diagnose the status of resolved-ts and safe-ts #15082 @ekexium
  • Expose some RocksDB configurations in TiKV that allow users to disable features such as TTL and periodic compaction #14873 @LykxSassinator
  • Avoid holding mutex when writing Titan manifest files to prevent affecting other threads #15351 @Connor1996
  • Optimize the compaction mechanism: when a Region is split, if there is no key to split, a compaction is triggered to eliminate excessive MVCC versions #15282 @SpadeA-Tang

• PD

  • Support blocking the Swagger API by default when the Swagger server is not enabled #6786 @bufferflies
  • Improve the high availability of etcd #6554 #6442 @lhy1024
  • Reduce the memory consumption of GetRegions requests #6835 @lhy1024
  • Support reusing HTTP connections #6913 @nolouch

• TiFlash

  • Improve TiFlash write performance through IO batch optimization #7735 @lidezhu
  • Improve TiFlash write performance by removing unnecessary fsync operations #7736 @lidezhu
  • Limit the maximum length of the TiFlash coprocessor task queue to avoid excessive queuing of coprocessor tasks, which impacts TiFlash's service availability #7747 @LittleFall

• Tools

  • Backup & Restore (BR)

    • Enhance support for connection reuse by setting MaxIdleConns and MaxIdleConnsPerHost parameters in the HTTP client #46011 @Leavrth
    • Improve fault tolerance of BR when it fails to connect to PD or external S3 storage #42909 @Leavrth
    • Add a new restore parameter WaitTiflashReady. When this parameter is enabled, the restore operation will be completed after TiFlash replicas are successfully replicated #43828 #46302 @3pointer

  • TiCDC

    • Refine the status message when TiCDC retries after a failure #9483 @asddongmen
    • Optimize the way TiCDC handles messages that exceed the limit when synchronizing to Kafka by supporting only sending the primary key to the downstream #9574 @3AceShowHand
    • Storage Sink now supports hexadecimal encoding for HEX formatted data, making it compatible with AWS DMS format specifications #9373 @CharlesCheung96

  • TiDB Data Migration (DM)

    • Support strict optimistic mode for incompatible DDL statements #9112 @GMHDBJD

  • Dumpling

    • Reduce export overhead by skipping querying all databases and tables when exporting data with the -sql parameter #45239 @lance6716

Bug fixes

• TiDB

  • Fix the issue that the index out of range error might be reported when pushing down the STREAM_AGG() operator #40857 @Dousir9
  • Fix the issue that TiDB ignores all partition information and creates a non-partitioned table when the CREATE TABLE statement contains sub-partition definitions #41198 #41200 @mjonss
  • Fix the issue that incorrect stale_read_ts setting might cause PREPARE stmt to read data incorrectly #43044 @you06
  • Fix the issue of possible data race in ActivateTxn #42092 @hawkingrei
  • Fix the issue that a batch client does not reconnect timely #44431 @crazycs520
  • Fix the issue that SQL compile error logs are not redacted #41831 @lance6716
  • Fix the issue that using CTEs and correlated subqueries simultaneously might result in incorrect query results or panic #44649 #38170 #44774 @winoros @guo-shaoge
  • Fix the issue that TTL tasks cannot trigger statistics updates in time #40109 @YangKeao
  • Fix the issue that the GC Resolve Locks step might miss some pessimistic locks #45134 @MyonKeminta
  • Fix the issue that memory leaks and duration continues to increase when connecting to TiDB using the binary protocol and executing a large number of PREPARE and EXECUTE statements #44612 @wshwsh12
  • Fix the issue that the data length in the QUERY column of the INFORMATION_SCHEMA.DDL_JOBS table might exceed the column definition #42440 @tiancaiamao
  • Fix the PD OOM issue when there is a large number of Regions but the table ID cannot be pushed down when querying some virtual tables using Prepare or Execute #39605 @djshow832
  • Fix the issue that statistics auto-collection might not trigger correctly on a partitioned table after adding a new index to the partitioned table #41638 @xuyifangreeneyes
  • Fix the issue that excessive memory consumption of SQL execution details in statistics causes TiDB OOM in extreme cases #44047 @wshwsh12
  • Fix the issue that batch coprocessor retry might generate incorrect Region information that causes query failure #44622 @windtalker
  • Fix the hang-up issue that occurs when queries with indexMerge are killed #45279 @xzhangxian1008
  • Fix the issue that querying the system table INFORMATION_SCHEMA.TIKV_REGION_STATUS returns incorrect results in some cases #45531 @Defined2014
  • Fix the issue that query results in MPP mode are incorrect when tidb_enable_parallel_apply is enabled #45299 @windtalker
  • Fix the issue that queries might return incorrect results when tidb_opt_agg_push_down is enabled #44795 @AilinKid
  • Fix the can't find proper physical plan issue caused by virtual columns #41014 @AilinKid
  • Fix the panic issue caused by empty processInfo #43829 @zimulala
  • Fix the issue that the query result of the SELECT CAST(n AS CHAR) statement is incorrect when n in the statement is a negative number #44786 @xhebox
  • Fix the issue that when the MySQL Cursor Fetch protocol is used, the memory consumption of result sets might exceed the tidb_mem_quota_query limit and causes TiDB OOM. After the fix, TiDB will automatically write result sets to the disk to release memory #43233 @YangKeao
  • Fix the duplicate entry error that occurs when restoring a table with AUTO_ID_CACHE=1 using BR #44716 @tiancaiamao
  • Fix the issue that the SELECT statement returns an error for a partitioned table if the table partition definition uses the FLOOR() function to round a partitioned column #42323 @jiyfhust
  • Fix the issue that concurrent view might cause DDL operations to be blocked #40352 @zeminzhou
  • Fix the issue that a statistics collection task fails due to an incorrect datetime value #39336 @xuyifangreeneyes
  • Fix the issue that some DDL statements might be stuck for a period after the PD node of a cluster is replaced #33908
  • Fix the issue that resolve lock might hang when there is a sudden change in PD time #44822 @zyguan
  • Fix the potential data race issue in index scan #45126 @wshwsh12
  • Fix the issue that the FormatSQL() method cannot properly truncate extremely long SQL statements in input #44542 @hawkingrei
  • Fix the issue that users can view information in the INFORMATION_SCHEMA.TIFLASH_REPLICA table even without permissions #45320 @Lloyd-Pottiger
  • Fix the issue that the behavior is inconsistent with MySQL when comparing a DATETIME or TIMESTAMP column with a number constant #38361 @yibin87
  • Fix the issue that an error in Index Join might cause the query to get stuck #45716 @wshwsh12
  • Fix the issue that killing a connection might cause go coroutine leaks #46034 @pingyu
  • Fix the issue that the tmp-storage-quota configuration does not take effect #45161 #26806 @wshwsh12
  • Fix the issue that TiFlash replicas might be unavailable when a TiFlash node is down in the cluster #38484 @hehechen
  • Fix the issue that TiDB crashes due to possible data race when reading and writing Config.Lables concurrently #45561 @genliqi
  • Fix the issue that the client-go regularly updating min-resolved-ts might cause PD OOM when the cluster is large #46664 @HuSharp

• TiKV

  • Fix the issue that the ttl-check-poll-interval configuration item does not take effect on RawKV API V2 #15142 @pingyu
  • Fix the issue that Online Unsafe Recovery does not abort on timeout #15346 @Connor1996
  • Fix the issue that Region Merge might be blocked after executing FLASHBACK #15258 @overvenus
  • Fix the data inconsistency issue that might occur when one TiKV node is isolated and another node is restarted #15035 @overvenus
  • Fix the issue that the QPS drops to zero in the sync-recover phase under the Data Replication Auto Synchronous mode #14975 @nolouch
  • Fix the issue that encryption might cause data corruption during partial write #15080 @tabokie
  • Fix the issue of heartbeat storms by reducing the number of store heartbeat retries #15184 @nolouch
  • Fix the issue that traffic control might not work when there is a high amount of pending compaction bytes #14392 @Connor1996
  • Fix the issue that network interruption between PD and TiKV might cause PITR to get stuck #15279 @YuJuncen
  • Fix the issue that TiKV might consume more memory when the Old Value feature in TiCDC is enabled #14815 @YuJuncen

• PD

  • Fix the issue that when etcd is already started but the client has not yet connected to it, calling the client might cause PD to panic #6860 @HuSharp
  • Fix the issue that a leader cannot exit for a long time #6918 @bufferflies
  • Fix the issue that when the placement rule uses LOCATION_LABLES, SQL and the Rule Checker are not compatible #38605 @nolouch
  • Fix the issue that PD might unexpectedly add multiple Learners to a Region #5786 @HunDunDM
  • Fix the issue that unhealthy peers cannot be removed when rule checker selects peers #6559 @nolouch
  • Fix the issue that failed learner peers in unsafe recovery are ignored in auto-detect mode #6690 @v01dstar

• TiFlash

  • Fix the issue that queries fail after fsp is changed for DATETIME, TIMESTAMP, or TIME data type #7809 @JaySon-Huang
  • Fix the issue that TiFlash data is inconsistent due to invalid range keys of a region #7762 @lidezhu
  • Fix the issue that when there are multiple HashAgg operators within the same MPP task, the compilation of the MPP task might take an excessively long time, severely affecting query performance #7810 @SeaRise
  • Fix the issue that TiFlash takes too long to restart after using Online Unsafe Recovery #7671 @hongyunyan
  • Fix the issue that TiFlash rounds the DECIMAL result incorrectly when doing division #6462 @LittleFall

• Tools

  • Backup & Restore (BR)

    • Fix the issue of restore failures by increasing the default values of the global parameters TableColumnCountLimit and IndexLimit used by BR to their maximum values #45793 @Leavrth
    • Fix the issue of rewrite failures when processing DDL meta information in PITR #43184 @Leavrth
    • Fix the issue of panic caused by not checking function returns during PITR execution #45853 @Leavrth
    • Fix the issue of obtaining invalid region ID when using S3-compatible storage other than Amazon S3 #41916 #42033 @3pointer
    • Fix the potential error in fine-grained backup phase in RawKV mode #37085 @pingyu
    • Fix the issue that the frequency of resolve lock is too high when there is no PITR backup task in the TiDB cluster #40759 @joccau
    • Alleviate the issue that the latency of the PITR log backup progress increases when Region leadership migration occurs #13638 @YuJuncen

  • TiCDC

    • Fix the issue that the replication task might get stuck when the downstream encounters an error and retries #9450 @hicqu
    • Fix the issue that the replication task fails due to short retry intervals when synchronizing to Kafka #9504 @3AceShowHand
    • Fix the issue that TiCDC might cause synchronization write conflicts when modifying multiple unique key rows in one transaction on the upstream #9430 @sdojjy
    • Fix the issue that TiCDC might incorrectly synchronize rename DDL operations #9488 #9378 #9531 @asddongmen
    • Fix the issue that the replication task might get stuck when the downstream encounters a short-term failure #9542 #9272#9582 #9592 @hicqu
    • Fix the panic issue that might occur when the TiCDC node status changes #9354 @sdojjy
    • Fix the issue that when Kafka Sink encounters errors it might indefinitely block changefeed progress #9309 @hicqu
    • Fix the issue that when the downstream is Kafka, TiCDC queries the downstream metadata too frequently and causes excessive workload in the downstream #8957 #8959 @hi-rustin
    • Fix the data inconsistency issue that might occur when some TiCDC nodes are isolated from the network #9344 @CharlesCheung96
    • Fix the issue that the replication task might get stuck when the redo log is enabled and there is an exception downstream #9172 @CharlesCheung96
    • Fix the issue that changefeeds would fail due to the temporary unavailability of PD #9294 @asddongmen
    • Fix the issue of too many downstream logs caused by frequently setting the downstream bidirectional replication-related variables when replicating data to TiDB or MySQL #9180 @asddongmen
    • Fix the issue that Avro protocol incorrectly identifies Enum type values #9259 @3AceShowHand

  • TiDB Data Migration (DM)

    • Fix the panic issue when the unique key column name is null #9247 @lance6716
    • Fix the deadlock issue that might occur when the validator handles errors incorrectly, and optimize the retry mechanism #9257 @D3Hunter
    • Fix the issue that collation is not considered when calculating the causality key #9489 @hihihuhu

  • TiDB Lightning

    • Fix the issue that the disk quota check might block when an engine is importing data #44867 @D3Hunter
    • Fix the issue that checksum reports an error Region is unavailable when SSL is enabled on the target cluster #45462 @D3Hunter
    • Fix the issue that encoding errors cannot be logged correctly #44321 @lyzx2001
    • Fix the issue that route might panic when importing CSV data #43284 @lyzx2001
    • Fix the issue that importing table A in logical import mode might mistakenly report that table B does not exist #44614 @dsdashun
    • Fix the issue that the data type is wrong when saving NEXT_GLOBAL_ROW_ID #45427 @lyzx2001
    • Fix the issue that checksum still reports errors when checksum = "optional" #45382 @lyzx2001
    • Fix the issue that data import fails when the PD cluster address changes #43436 @lichunzhu
    • Fix the issue that data import fails when some PD nodes fail #43400 @lichunzhu
    • Fix the issue that when a table with auto-incremental columns sets AUTO_ID_CACHE=1, the base value of the ID allocator is incorrect #46100 @D3Hunter

  • Dumpling

    • Fix the issue that the exported file is lost due to unprocessed file writer closing error when exporting to Amazon S3 #45353 @lichunzhu

  • TiDB Binlog

    • Fix the issue that the etcd client does not automatically synchronize the latest node information during initialization #1236 @lichunzhu