TiDB 7.5.1 Release Notes

Release date: February 29, 2024

TiDB version: 7.5.1

Quick access: Quick start | Production deployment

Compatibility changes

Prohibit setting require_secure_transport to ON in Security Enhanced Mode (SEM) to prevent potential connectivity issues for users #47665 @tiancaiamao
To reduce the overhead of log printing, TiFlash changes the default value of logger.level from "debug" to "info" #8641 @JaySon-Huang
Introduce the TiKV configuration item gc.num-threads to set the number of GC threads when enable-compaction-filter is false #16101 @tonyxuqqi
TiCDC Changefeed introduces the following new configuration items:
- compression: enables you to configure the compression behavior of redo log files #10176 @sdojjy
- sink.cloud-storage-config: enables you to set the automatic cleanup of historical data when replicating data to object storage #10109 @CharlesCheung96
- consistent.flush-concurrency: enables you to set the concurrency for uploading a single redo file #10226 @sdojjy

Improvements

TiDB
- Use tikv_client_read_timeout during the DDL schema reload process to reduce the impact of Meta Region Leader read unavailability on the cluster #48124 @cfzjywxk
- Enhance observability related to resource control #49318 @glorv @bufferflies @nolouch
  As more and more users use resource groups to isolate application workloads, Resource Control provides enhanced data based on resource groups. This helps you monitor resource group workloads and settings, ensuring that you can quickly identify and accurately diagnose problems, including:
  - Slow Queries: add the resource group name, resource unit (RU) consumption, and time for waiting for resources.
  - Statement Summary Tables: add the resource group name, RU consumption, and time for waiting for resources.
  - In the system variable tidb_last_query_info, add a new entry ru_consumption to indicate the consumed RU by SQL statements. You can use this variable to get the resource consumption of the last statement in the session.
  - Add database metrics based on resource groups: QPS/TPS, execution time (P999/P99/P95), number of failures, and number of connections.
- Modify the CANCEL IMPORT JOB statement to a synchronous statement #48736 @D3Hunter
- Support the FLASHBACK CLUSTER TO TSO syntax #48372 @BornChanger
- Optimize the TiDB implementation when handling some type conversions and fix related issues #47945 #47864 #47829 #47816 @YangKeao @lcwangchao
- When a non-binary collation is set and the query includes LIKE, the optimizer generates an IndexRangeScan to improve the execution efficiency #48181 #49138 @time-and-fate
- Enhance the ability to convert OUTER JOIN to INNER JOIN in specific scenarios #49616 @qw4990
- Support multiple accelerated ADD INDEX DDL tasks to be queued for execution, instead of falling back to normal ADD INDEX tasks #47758 @tangenta
- Improve the speed of adding indexes to empty tables #49682 @zimulala
TiFlash
- Improve the calculation method for Request Unit (RU) to make RU values more stable #8391 @guo-shaoge
- Reduce the impact of disk performance jitter on read latency #8583 @JaySon-Huang
- Reduce the impact of background GC tasks on read and write task latency #8650 @JaySon-Huang
Tools
- Backup & Restore (BR)
  - Improve the speed of merging SST files during data restore by using a more efficient algorithm #50613 @Leavrth
  - Support creating databases in batch during data restore #50767 @Leavrth
  - Support ingesting SST files in batch during data restore #16267 @3pointer
  - Print the information of the slowest Region that affects global checkpoint advancement in logs and metrics during log backups #51046 @YuJuncen
  - Improve the table creation performance of the RESTORE statement in scenarios with large datasets #48301 @Leavrth
  - BR can pause Region merging by setting the merge-schedule-limit configuration to 0 #7148 @BornChanger
  - Refactor the BR exception handling mechanism to increase tolerance for unknown errors #47656 @3pointer
- TiCDC
  - Support searching TiCDC logs in the TiDB Dashboard #10263 @CharlesCheung96
  - Support querying the downstream synchronization status of a changefeed, which helps you determine whether the upstream data changes received by TiCDC have been synchronized to the downstream system completely #10289 @hongyunyan
  - Improve the performance of TiCDC replicating data to object storage by increasing parallelism #10098 @CharlesCheung96
- TiDB Lightning
  - Improve the performance of ALTER TABLE when importing a large number of small tables #50105 @D3Hunter

Bug fixes

TiDB
- Fix the issue that setting the system variable tidb_service_scope does not take effect #49245 @ywqzzy
- Fix the issue that the communication protocol cannot handle packets larger than or equal to 16 MB when compression is enabled #47157 #47161 @dveeden
- Fix the issue that the approx_percentile function might cause TiDB panic #40463 @xzhangxian1008
- Fix the issue that TiDB might implicitly insert the from_binary function when the argument of a string function is a NULL constant, causing some expressions unable to be pushed down to TiFlash #49526 @YangKeao
- Fix the goroutine leak issue that might occur when the HashJoin operator fails to spill to disk #50841 @wshwsh12
- Fix the issue that BIT type columns might cause query errors due to decode failures when they are involved in calculations of some functions #49566 #50850 #50855 @jiyfhust
- Fix the goroutine leak issue that occurs when the memory usage of CTE queries exceed limits #50337 @guo-shaoge
- Fix the issue that wrong results might be returned when TiFlash late materialization processes associated columns #49241 #51204 @Lloyd-Pottiger
- Fix the issue that the background job thread of TiDB might panic when TiDB records historical statistics #49076 @hawkingrei
- Fix the error that might occur when TiDB merges histograms of global statistics for partitioned tables #49023 @hawkingrei
- Fix the issue that the historical statistics of the stats_meta table are not updated after a partition is dropped #49334 @hi-rustin
- Fix the issue of incorrect query results caused by multi-valued indexes mistakenly selected as the Index Join probe side #50382 @AilinKid
- Fix the issue that the USE_INDEX_MERGE hint does not take effect on multi-valued indexes #50553 @AilinKid
- Fix the issue that users might get errors when querying the INFORMATION_SCHEMA.ANALYZE_STATUS system table #48835 @hi-rustin
- Fix the issue of wrong query results due to TiDB incorrectly eliminating constant values in group by #38756 @hi-rustin
- Fix the issue that the processed_rows of the ANALYZE task on a table might exceed the total number of rows in that table #50632 @hawkingrei
- Fix the issue that TiDB might panic when using the EXECUTE statement to execute PREPARE STMT after the tidb_enable_prepared_plan_cache system variable is enabled and then disabled #49344 @qw4990
- Fix the Column ... in from clause is ambiguous error that might occur when a query uses NATURAL JOIN #32044 @AilinKid
- Fix the issue that using a multi-valued index to access an empty JSON array might return incorrect results #50125 @YangKeao
- Fix the Can't find column ... error that might occur when aggregate functions are used for group calculations #50926 @qw4990
- Fix the issue that the control of SET_VAR for variables of the string type might become invalid #50507 @qw4990
- Fix the issue that high CPU usage of TiDB occurs due to long-term memory pressure caused by tidb_server_memory_limit #48741 @XuHuaiyu
- Fix the issue that the completion times of two DDL tasks with dependencies are incorrectly sequenced #49498 @tangenta
- Fix the issue that illegal optimizer hints might cause valid hints to be ineffective #49308 @hawkingrei
- Fix the issue that DDL statements with the CHECK constraint are stuck #47632 @jiyfhust
- Fix the issue that the behavior of the ENFORCED option in the CHECK constraint is inconsistent with MySQL 8.0 #47567 #47631 @jiyfhust
- Fix the issue that CTE queries might report an error type assertion for CTEStorageMap failed during the retry process #46522 @tiancaiamao
- Fix the issue that the DELETE and UPDATE statements using index lookup might report an error when tidb_multi_statement_mode mode is enabled #50012 @tangenta
- Fix the issue that UPDATE or DELETE statements containing WITH RECURSIVE CTEs might produce incorrect results #48969 @winoros
- Fix the issue that the optimizer incorrectly converts TiFlash selection path to the DUAL table in specific scenarios #49285 @AilinKid
- Fix the issue that the same query plan has different PLAN_DIGEST values in some cases #47634 @King-Dylan
- Fix the issue that after the time window for automatic statistics updates is configured, statistics might still be updated outside that time window #49552 @hawkingrei
- Fix the issue that the query result is incorrect when an ENUM type column is used as the join key #48991 @winoros
- Fix the issue that executing UNIQUE index lookup with an ORDER BY clause might cause an error #49920 @jackysp
- Fix the issue that LIMIT in multi-level nested UNION queries might become ineffective #49874 @Defined2014
- Fix the issue that the result of COUNT(INT) calculated by MPP might be incorrect #48643 @AilinKid
- Fix the issue that parsing invalid values of ENUM or SET types would directly cause SQL statement errors #49487 @winoros
- Fix the issue that TiDB panics and reports an error invalid memory address or nil pointer dereference #42739 @CbcWestwolf
- Fix the issue that executing UNION ALL with the DUAL table as the first subnode might cause an error #48755 @winoros
- Fix the issue that common hints do not take effect in UNION ALL statements #50068 @hawkingrei
- Fix the issue that TiDB server might panic during graceful shutdown #36793 @bb7133
- Fix the issue that Daylight Saving Time is displayed incorrectly in some time zones #49586 @overvenus
- Fix the issue that static CALIBRATE RESOURCE relies on the Prometheus data #49174 @glorv
- Fix the issue that hints cannot be used in REPLACE INTO statements #34325 @YangKeao
- Fix the issue that executing queries containing the GROUP_CONCAT(ORDER BY) syntax might return errors #49986 @AilinKid
- Fix the issue that TiDB server might consume a significant amount of resources when the enterprise plugin for audit logging is used #49273 @lcwangchao
- Fix the issue that using old interfaces might cause inconsistent metadata for tables #49751 @hawkingrei
- Fix the issue that disabling tidb_enable_collect_execution_info causes the coprocessor cache to panic #48212 @you06
- Fix the issue that executing ALTER TABLE ... LAST PARTITION fails when the partition column type is DATETIME #48814 @crazycs520
- Fix the issue that the COMMIT or ROLLBACK operation executed through COM_STMT_EXECUTE fails to terminate transactions that have timed out #49151 @zyguan
- Fix the issue that histogram statistics might not be parsed into readable strings when the histogram boundary contains NULL #49823 @AilinKid
- Fix the issue that queries containing common table expressions (CTEs) unexpectedly get stuck when the memory limit is exceeded #49096 @AilinKid
- Fix the issue that data is inconsistent under the TiDB Distributed eXecution Framework (DXF) when executing ADD INDEX after the DDL Owner is network isolated #49773 @tangenta
- Fix the issue that the auto-increment ID allocation reports an error due to concurrent conflicts when using an auto-increment column with AUTO_ID_CACHE=1 #50519 @tiancaiamao
- Fix the issue that TiDB might panic when a query contains the Apply operator and the fatal error: concurrent map writes error occurs #50347 @SeaRise
- Fix the TiDB node panic issue that occurs when DDL jobID is restored to 0 #46296 @jiyfhust
- Fix the issue that query results are incorrect due to STREAM_AGG() incorrectly handling CI #49902 @wshwsh12
- Mitigate the issue that TiDB nodes might encounter OOM errors when dealing with a large number of tables or partitions #50077 @zimulala
- Fix the issue that the LEADING hint does not take effect in UNION ALL statements #50067 @hawkingrei
- Fix the issue that LIMIT and OPRDERBY might be invalid in nested UNION queries #49377 @AilinKid
- Fix the issue that a query containing the IndexHashJoin operator gets stuck when memory exceeds tidb_mem_quota_query #49033 @XuHuaiyu
- Fix the issue that TiDB returns wrong query results when processing ENUM or SET types by constant propagation #49440 @winoros
- Fix the issue that executing SELECT INTO OUTFILE using the PREPARE method incorrectly returns a success message instead of an error #49166 @qw4990
- Fix the issue that enforced sorting might become ineffective when a query uses optimizer hints (such as STREAM_AGG()) that enforce sorting and its execution plan contains IndexMerge #49605 @AilinKid
- Fix the issue that tables with AUTO_ID_CACHE=1 might lead to gRPC client leaks when there are a large number of tables #48869 @tiancaiamao
- Fix the issue that in non-strict mode (sql_mode = ''), truncation during executing INSERT still reports an error #49369 @tiancaiamao
- Fix the issue that using the _ wildcard in LIKE when the data contains trailing spaces can result in incorrect query results #48983 @time-and-fate
- Fix the issue that executing ADMIN CHECK after updating the tidb_mem_quota_query system variable returns ERROR 8175 #49258 @tangenta
- Fix the issue of excessive statistical error in constructing statistics caused by Golang's implicit conversion algorithm #49801 @qw4990
- Fix the issue that queries containing CTEs report runtime error: index out of range [32] with length 32 when tidb_max_chunk_size is set to a small value #48808 @guo-shaoge
TiKV
- Fix the issue that enabling tidb_enable_row_level_checksum might cause TiKV to panic #16371 @cfzjywxk
- Fix the issue that TiKV might panic when gRPC threads are checking is_shutdown #16236 @pingyu
- Fix the issue that TiKV converts the time zone incorrectly for Brazil and Egypt #16220 @overvenus
- Fix the issue that blob-run-mode in Titan cannot be updated online #15978 @tonyxuqqi
- Fix the issue that TiDB and TiKV might produce inconsistent results when processing DECIMAL arithmetic multiplication truncation #16268 @solotzg
- Fix the issue that Flashback might get stuck when encountering notLeader or regionNotFound #15712 @HuSharp
- Fix the issue that damaged SST files might be spread to other TiKV nodes #15986 @Connor1996
- Fix the issue that if TiKV runs extremely slowly, it might panic after Region merge #16111 @overvenus
- Fix the issue that the joint state of DR Auto-Sync might time out when scaling out #15817 @Connor1996
- Fix the issue that Resolved TS might be blocked for two hours #11847 #15520 #39130 @overvenus
- Fix the issue that cast_duration_as_time might return incorrect results #16211 @gengliqi
PD
- Fix the issue that querying resource groups in batch might cause PD to panic #7206 @nolouch
- Fix the issue that PD cannot read resource limitations when it is started with systemd #7628 @bufferflies
- Fix the issue that continuous jitter in PD disk latency might cause PD to fail to select a new leader #7251 @HuSharp
- Fix the issue that a network partition in PD might cause scheduling not to be started immediately #7016 @HuSharp
- Fix the issue that the PD monitoring item learner-peer-count does not synchronize the old value after a leader switch #7728 @CabinfeverB
- Fix the issue that when PD leader is transferred and there is a network partition between the new leader and the PD client, the PD client fails to update the information of the leader #7416 @CabinfeverB
- Fix some security issues by upgrading the version of Gin Web Framework from v1.8.1 to v1.9.1 #7438 @niubell
- Fix the issue that the orphan peer is deleted when the number of replicas does not meet the requirements #7584 @bufferflies
- Fix the issue that querying a Region without a leader using pd-ctl might cause PD to panic #7630 @rleungx
TiFlash
- Fix the issue that TiFlash might panic due to unstable network connections with PD during replica migration #8323 @JaySon-Huang
- Fix the issue that removing and then re-adding TiFlash replicas might lead to data corruption in TiFlash #8695 @JaySon-Huang
- Fix a potential issue that FLASHBACK TABLE or RECOVER TABLE might fail to recover data of some TiFlash replicas if DROP TABLE is executed immediately after data insertion #8395 @JaySon-Huang
- Fix incorrect display of maximum percentile time for some panels in Grafana #8076 @JaySon-Huang
- Fix the issue that TiFlash might crash during remote reads #8685 @guo-shaoge
- Fix the issue that TiFlash incorrectly handles ENUM when the ENUM value is 0 #8311 @solotzg
- Fix the issue that short queries executed successfully print excessive info logs #8592 @windtalker
- Fix the issue that the memory usage increases significantly due to slow queries #8564 @JinheLin
- Fix the issue that the lowerUTF8 and upperUTF8 functions do not allow characters in different cases to occupy different bytes #8484 @gengliqi
- Fix the potential OOM issue that might occur when scanning multiple partitioned tables during stream read #8505 @gengliqi
- Fix the issue of memory leak when TiFlash encounters memory limitation during query #8447 @JinheLin
- Fix the TiFlash panic issue when TiFlash encounters conflicts during concurrent DDL execution #8578 @JaySon-Huang
- Fix the issue that TiFlash panics after executing ALTER TABLE ... MODIFY COLUMN ... NOT NULL, which changes nullable columns to non-nullable #8419 @JaySon-Huang
- Fix the issue that query results are incorrect when querying with filtering conditions like ColumnRef in (Literal, Func...) #8631 @Lloyd-Pottiger
- Fix the issue that data of TiFlash replicas would still be garbage collected after executing FLASHBACK DATABASE #8450 @JaySon-Huang
- Fix the issue that TiFlash might not be able to select the GC owner of object storage data under the disaggregated storage and compute architecture #8519 @JaySon-Huang
- Fix the random invalid memory access issue that might occur with GREATEST or LEAST functions containing constant string parameters #8604 @windtalker
- Fix the issue that TiFlash replica data might be accidentally deleted after performing point-in-time recovery (PITR) or executing FLASHBACK CLUSTER TO, which might result in data anomalies #8777 @JaySon-Huang
- Fix the issue that TiFlash Anti Semi Join might return incorrect results when the join includes non-equivalent conditions #8791 @windtalker
Tools
- Backup & Restore (BR)
  - Fix the issue that data restore is slowed down due to absence of a leader on a TiKV node #50566 @Leavrth
  - Fix the issue that full restore still requires the target cluster to be empty after the --filter option is specified #51009 @3pointer
  - Fix the issue that when resuming from a checkpoint after data restore fails, an error the target cluster is not fresh occurs #50232 @Leavrth
  - Fix the issue that stopping a log backup task causes TiDB to crash #50839 @YuJuncen
  - Fix the issue that the Unsupported collation error is reported when you restore data from backups of an old version #49466 @3pointer
  - Fix the issue that the log backup task can start but does not work properly if failing to connect to PD during task initialization #16056 @YuJuncen
  - Fix the issue that BR generates incorrect URIs for external storage files #48452 @3AceShowHand
  - Fix the issue that log backup gets stuck after changing the TiKV IP address on the same node #50445 @3pointer
  - Fix the issue that BR cannot retry when encountering an error while reading file content from S3 #49942 @Leavrth
- TiCDC
  - Fix the issue that the sink module fails to restart correctly after encountering an error when Syncpoint is enabled (enable-sync-point = true) #10091 @hicqu
  - Fix the issue that the file sequence number generated by the storage service might not increment correctly when using the storage sink #10352 @CharlesCheung96
  - Fix the issue that the Syncpoint table might be incorrectly replicated #10576 @asddongmen
  - Fix the issue that OAuth2.0, TLS, and mTLS cannot be enabled properly when using Apache Pulsar as the downstream #10602 @asddongmen
  - Fix the issue that TiCDC returns the ErrChangeFeedAlreadyExists error when concurrently creating multiple changefeeds #10430 @CharlesCheung96
  - Fix the issue that the changefeed resolved ts does not advance in extreme cases #10157 @sdojjy
  - Fix the issue that TiCDC mistakenly closes the connection with TiKV in certain special scenarios #10239 @hicqu
  - Fix the issue that the TiCDC server might panic when replicating data to an object storage service #10137 @sdojjy
  - Fix the issue that the changefeed reports an error after TRUNCATE PARTITION is executed on the upstream table #10522 @sdojjy
  - Fix the issue that after filtering out add table partition events is configured in ignore-event, TiCDC does not replicate other types of DML changes for related partitions to the downstream #10524 @CharlesCheung96
  - Fix the potential data race issue during kv-client initialization #10095 @3AceShowHand
- TiDB Data Migration (DM)
  - Fix the issue that a migration task error occurs when the downstream table structure contains shard_row_id_bits #10308 @GMHDBJD
  - Fix the issue that DM encounters "event type truncate not valid" error that causes the upgrade to fail #10282 @GMHDBJD

TiDB 7.5.1 Release Notes

Compatibility changes

Improvements

Bug fixes

Was this page helpful?