Important

You are viewing the documentation of an older version of the TiDB database (TiDB v5.3). It is recommended that you use the latest LTS version of the TiDB database.

Introduction to Statistics

TiDB uses statistics to decide which index to choose. The tidb_analyze_version variable controls the statistics collected by TiDB. Currently, two versions of statistics are supported: tidb_analyze_version = 1 and tidb_analyze_version = 2. In versions before v5.1.0, the default value of this variable is 1. In v5.3.0 and later versions, the default value of this variable is 2, which serves as an experimental feature. If your cluster is upgraded from a version earlier than v5.3.0 to v5.3.0 or later, the default value of tidb_analyze_version does not change.

Note

When tidb_analyze_version = 2, if memory overflow occurs after ANALYZE is executed, you need to set tidb_analyze_version = 1 and perform one of the following operations:

If the ANALYZE statement is executed manually, manually analyze every table to be analyzed.

select distinct(concat('ANALYZE ',table_schema, '.', table_name,';')) from information_schema.tables, mysql.stats_histograms where stats_ver = 2 and table_id = tidb_table_id ;

If TiDB automatically executes the ANALYZE statement because the auto-analysis has been enabled, execute the following statement that generates the DROP STATS statement:

select distinct(concat('DROP STATS ',table_schema, '.', table_name,';')) from information_schema.tables, mysql.stats_histograms where stats_ver = 2 and table_id = tidb_table_id ;

If the result of the preceding statement is too long to copy and paste, you can export the result to a temporary text file and then perform execution from the file like this:
```
SELECT DISTINCT ... INTO OUTFILE '/tmp/sql.txt';
mysql -h XXX -u user -P 4000 ... < '/tmp/sql.txt';
```

These two versions include different information in TiDB:

Information	Version 1	Version 2
The total number of rows in the table	√	√
Column Count-Min Sketch	√	×
Index Count-Min Sketch	√	×
Column Top-N	√	√ (Maintenance methods and precision are improved)
Index Top-N	√ (Insufficient maintenance precision might cause inaccuracy)	√ (Maintenance methods and precision are improved)
Column histogram	√	√ (The histogram does not include Top-N values.)
Index histogram	√	√ (The histogram buckets record the number of different values in each bucket, and the histogram does not include Top-N values.)
The number of `NULL`s in the column	√	√
The number of `NULL`s in the index	√	√
The average length of columns	√	√
The average length of indexes	√	√

Compared to Version 1, Version 2 statistics avoids the potential inaccuracy caused by hash collision when the data volume is huge. It also maintains the estimate precision in most scenarios.

This document briefly introduces the histogram, Count-Min Sketch, and Top-N, and details the collection and maintenance of statistics.

Histogram

A histogram is an approximate representation of the distribution of data. It divides the entire range of values into a series of buckets, and uses simple data to describe each bucket, such as the number of values falling in the bucket. In TiDB, an equal-depth histogram is created for the specific columns of each table. The equal-depth histogram can be used to estimate the interval query.

Here "equal-depth" means that the number of values falling into each bucket is as equal as possible. For example, for a given set {1.6, 1.9, 1.9, 2.0, 2.4, 2.6, 2.7, 2.7, 2.8, 2.9, 3.4, 3.5}, you want to generate 4 buckets. The equal-depth histogram is as follows. It contains four buckets [1.6, 1.9], [2.0, 2.6], [2.7, 2.8], [2.9, 3.5]. The bucket depth is 3.

Equal-depth Histogram Example

For details about the parameter that determines the upper limit to the number of histogram buckets, refer to Manual Collection. When the number of buckets is larger, the accuracy of the histogram is higher; however, higher accuracy is at the cost of the usage of memory resources. You can adjust this number appropriately according to the actual scenario.

Count-Min Sketch

Count-Min Sketch is a hash structure. When an equivalence query contains a = 1 or IN query (for example, a in (1, 2, 3)), TiDB uses this data structure for estimation.

A hash collision might occur since Count-Min Sketch is a hash structure. In the EXPLAIN statement, if the estimate of the equivalent query deviates greatly from the actual value, it can be considered that a larger value and a smaller value have been hashed together. In this case, you can take one of the following ways to avoid the hash collision:

Modify the WITH NUM TOPN parameter. TiDB stores the high-frequency (top x) data separately, with the other data stored in Count-Min Sketch. Therefore, to prevent a larger value and a smaller value from being hashed together, you can increase the value of WITH NUM TOPN. In TiDB, its default value is 20. The maximum value is 1024. For more information about this parameter, see Full Collection.
Modify two parameters WITH NUM CMSKETCH DEPTH and WITH NUM CMSKETCH WIDTH. Both affect the number of hash buckets and the collision probability. You can increase the values of the two parameters appropriately according to the actual scenario to reduce the probability of hash collision, but at the cost of higher memory usage of statistics. In TiDB, the default value of WITH NUM CMSKETCH DEPTH is 5, and the default value of WITH NUM CMSKETCH WIDTH is 2048. For more information about the two parameters, see Full Collection.

Top-N values

Top-N values are values with the top N occurrences in a column or index. TiDB records the values and occurrences of Top-N values.

Collect statistics

Manual collection

You can run the ANALYZE statement to collect statistics.

Note

The execution time of ANALYZE TABLE in TiDB is longer than that in MySQL or InnoDB. In InnoDB, only a small number of pages are sampled, while in TiDB a comprehensive set of statistics is completely rebuilt. Scripts that were written for MySQL may naively expect ANALYZE TABLE will be a short-lived operation.

For quicker analysis, you can set tidb_enable_fast_analyze to 1 to enable the Quick Analysis feature. The default value for this parameter is 0.

After Quick Analysis is enabled, TiDB randomly samples approximately 10,000 rows of data to build statistics. Therefore, in the case of uneven data distribution or a relatively small amount of data, the accuracy of statistical information is relatively poor. It might lead to poor execution plans, such as choosing the wrong index. If the execution time of the normal ANALYZE statement is acceptable, it is recommended to disable the Quick Analysis feature.

tidb_enable_fast_analyze is an experimental feature, which currently does not match exactly with the statistical information of tidb_analyze_version=2. Therefore, you need to set the value of tidb_analyze_version to 1 when tidb_enable_fast_analyze is enabled.

Full collection

You can perform full collection using the following syntax.

To collect statistics of all the tables in TableNameList:

ANALYZE TABLE TableNameList [WITH NUM BUCKETS|TOPN|CMSKETCH DEPTH|CMSKETCH WIDTH]|[WITH NUM SAMPLES|WITH FLOATNUM SAMPLERATE];

WITH NUM BUCKETS specifies the maximum number of buckets in the generated histogram.
WITH NUM TOPN specifies the maximum number of the generated TOPNs.
WITH NUM CMSKETCH DEPTH specifies the depth of the CM Sketch.
WITH NUM CMSKETCH WIDTH specifies the width of the CM Sketch.
WITH NUM SAMPLES specifies the number of samples.
WITH FLOAT_NUM SAMPLERATE specifies the sampling rate.

WITH NUM SAMPLES and WITH FLOAT_NUM SAMPLERATE correspond to two different algorithms of collecting samples.

WITH NUM SAMPLES specifies the size of the sampling set, which is implemented in the reservoir sampling method in TiDB. When a table is large, it is not recommended to use this method to collect statistics. Because the intermediate result set of the reservoir sampling contains redundant results, it causes additional pressure on resources such as memory.
WITH FLOAT_NUM SAMPLERATE is a sampling method introduced in v5.3.0. With the value range (0, 1], this parameter specifies the sampling rate. It is implemented in the way of Bernoulli sampling in TiDB, which is more suitable for sampling larger tables and performs better in collection efficiency and resource usage.

Before v5.3.0, TiDB uses the reservoir sampling method to collect statistics. Since v5.3.0, the TiDB Version 2 statistics uses the Bernoulli sampling method to collect statistics by default. To re-use the reservoir sampling method, you can use the WITH NUM SAMPLES statement.

Note

The current sampling rate is calculated based on an adaptive algorithm. When you can observe the number of rows in a table using SHOW STATS_META, you can use this number of rows to calculate the sampling rate corresponding to 100,000 rows. If you cannot observe this number, you can use the TABLE_KEYS column in the TABLE_STORAGE_STATS table as another reference to calculate the sampling rate.

Normally, STATS_META is more credible than TABLE_KEYS. However, after importing data through the methods like TiDB Lightning local-backend, the result of STATS_META is 0. To handle this situation, you can use TABLE_KEYS to calculate the sampling rate when the result of STATS_META is much smaller than the result of TABLE_KEYS.

The following syntax collects statistics for some columns in the TableName table:

ANALYZE TABLE TableName COLUMNS ColumnNameList [WITH NUM BUCKETS|TOPN|CMSKETCH DEPTH|CMSKETCH WIDTH]|[WITH NUM SAMPLES|WITH FLOATNUM SAMPLERATE];

This syntax collects statistics on the specified columns and indexes, as well as the statistics on the columns involved in the extended statistics. If the number of columns in the table is large, the columns that require statistics might only be a small subset of the table. In this situation, this syntax can greatly reduce the stress of collecting statistics.

Note

The syntax above takes effect only when tidb_analyze_version = 2.
In the syntax above, ColumnNameList cannot be empty.
The syntax above collects the full statistics of a table. For example, after collecting the statistics of column a and column b, to further collect the statistics of column c, you need to specify all three columns in the statement ANALYZE table t columns a, b, c rather than specifying only the additional column c like ANALYZE TABLE t COLUMNS c.

To collect statistics of the index columns on all IndexNameLists in TableName:
```
ANALYZE TABLE TableName INDEX [IndexNameList] [WITH NUM BUCKETS|TOPN|CMSKETCH DEPTH|CMSKETCH WIDTH|SAMPLES]|[WITH FLOATNUM SAMPLERATE];
```
The statement collects statistics of all index columns when IndexNameList is empty.

To collect statistics of partition in all PartitionNameLists in TableName:

ANALYZE TABLE TableName PARTITION PartitionNameList [WITH NUM BUCKETS|TOPN|CMSKETCH DEPTH|CMSKETCH WIDTH]|[WITH NUM SAMPLES|WITH FLOATNUM SAMPLERATE];

To collect statistics of some columns for the partitions in all PartitionNameLists in TableName:

ANALYZE TABLE TableName PARTITION PartitionNameList COLUMNS ColumnNameList [WITH NUM BUCKETS|TOPN|CMSKETCH DEPTH|CMSKETCH WIDTH]|[WITH NUM SAMPLES|WITH FLOATNUM SAMPLERATE];

To collect statistics of index columns for the partitions in all PartitionNameLists in TableName:

ANALYZE TABLE TableName PARTITION PartitionNameList INDEX [IndexNameList] [WITH NUM BUCKETS|TOPN|CMSKETCH DEPTH|CMSKETCH WIDTH]|[WITH NUM SAMPLES|WITH FLOATNUM SAMPLERATE];

Note

To ensure that the statistical information before and after the collection is consistent, when you set tidb_analyze_version=2, ANALYZE TABLE TableName INDEX will also collect statistics of the whole table instead of the given index.

Incremental collection

To improve the speed of analysis after full collection, incremental collection could be used to analyze the newly added sections in monotonically non-decreasing columns such as time columns.

Note

Currently, the incremental collection is only provided for index.
When using the incremental collection, you must ensure that only INSERT operations exist on the table, and that the newly inserted value on the index column is monotonically non-decreasing. Otherwise, the statistical information might be inaccurate, affecting the TiDB optimizer to select an appropriate execution plan.

You can perform incremental collection using the following syntax.

To incrementally collect statistics for index columns in all IndexNameLists in TableName:

ANALYZE INCREMENTAL TABLE TableName INDEX [IndexNameList] [WITH NUM BUCKETS|TOPN|CMSKETCH DEPTH|CMSKETCH WIDTH]|[WITH NUM SAMPLES|WITH FLOATNUM SAMPLERATE];

To incrementally collect statistics of index columns for partitions in all PartitionNameLists in TableName:

ANALYZE INCREMENTAL TABLE TableName PARTITION PartitionNameList INDEX [IndexNameList] [WITH NUM BUCKETS|TOPN|CMSKETCH DEPTH|CMSKETCH WIDTH]|[WITH NUM SAMPLES|WITH FLOATNUM SAMPLERATE];

Automatic update

For the INSERT, DELETE, or UPDATE statements, TiDB automatically updates the number of rows and updated rows. TiDB persists this information regularly and the update cycle is 20 * stats-lease. The default value of stats-lease is 3s. If you specify the value as 0, it does not update automatically.

Three system variables related to automatic update of statistics are as follows:

System Variable	Default Value	Description
`tidb_auto_analyze_ratio`	0.5	The threshold value of automatic update
`tidb_auto_analyze_start_time`	`00:00 +0000`	The start time in a day when TiDB can perform automatic update
`tidb_auto_analyze_end_time`	`23:59 +0000`	The end time in a day when TiDB can perform automatic update

When the ratio of the number of modified rows to the total number of rows of tbl in a table is greater than tidb_auto_analyze_ratio, and the current time is between tidb_auto_analyze_start_time and tidb_auto_analyze_end_time, TiDB executes the ANALYZE TABLE tbl statement in the background to automatically update the statistics of this table.

Note

Currently, the automatic update does not record the configuration items input at manual ANALYZE. Therefore, when you use the WITH syntax to control the collecting behavior of ANALYZE, you need to manually set scheduled tasks to collect statistics.

Before v5.0, when the query is executed, TiDB collects feedback with the probability of feedback-probability and uses it to update the histogram and Count-Min Sketch. In the current version, this feature is experimental and disabled by default, and it is not recommended to enable this feature in the production environment.

Control `ANALYZE` concurrency

When you run the ANALYZE statement, you can adjust the concurrency using the following parameters, to control its effect on the system.

`tidb_build_stats_concurrency`

Currently, when you run the ANALYZE statement, the task is divided into multiple small tasks. Each task only works on one column or index. You can use the tidb_build_stats_concurrency parameter to control the number of simultaneous tasks. The default value is 4.

`tidb_distsql_scan_concurrency`

When you analyze regular columns, you can use the tidb_distsql_scan_concurrency parameter to control the number of Region to be read at one time. The default value is 15.

`tidb_index_serial_scan_concurrency`

When you analyze index columns, you can use the tidb_index_serial_scan_concurrency parameter to control the number of Region to be read at one time. The default value is 1.

View `ANALYZE` state

When executing the ANALYZE statement, you can view the current state of ANALYZE using the following SQL statement:

SHOW ANALYZE STATUS [ShowLikeOrWhere]

This statement returns the state of ANALYZE. You can use ShowLikeOrWhere to filter the information you need.

Currently, the SHOW ANALYZE STATUS statement returns the following 7 columns:

Syntax Element	Description
table_schema	The database name
table_name	The table name
partition_name	The partition name
job_info	The task information. The element includes index names when index analysis is performed.
row_count	The number of rows that have been analyzed
start_time	The time at which the task starts
state	The state of a task, including `pending`, `running`, `finished`, and `failed`

View statistics

You can view the statistics status using the following statements.

Metadata of tables

You can use the SHOW STATS_META statement to view the total number of rows and the number of updated rows.

The syntax of ShowLikeOrWhereOpt is as follows:

SHOW STATS_META [ShowLikeOrWhere]

Currently, the SHOW STATS_META statement returns the following 6 columns:

Syntax Element	Description
`db_name`	The database name
`table_name`	The table name
`partition_name`	The partition name
`update_time`	The time of the update
`modify_count`	The number of modified rows
`row_count`	The total number of rows

Note

When TiDB automatically updates the total number of rows and the number of modified rows according to DML statements, update_time is also updated. Therefore, update_time does not necessarily indicate the last time when the ANALYZE statement is executed.

Health state of tables

You can use the SHOW STATS_HEALTHY statement to check the health state of tables and roughly estimate the accuracy of the statistics. When modify_count >= row_count, the health state is 0; when modify_count < row_count, the health state is (1 - modify_count/row_count) * 100.

The synopsis of SHOW STATS_HEALTHY is:

ShowStatsHealthy

and the synopsis of the ShowLikeOrWhereOpt part is:

ShowLikeOrWhereOpt

Currently, the SHOW STATS_HEALTHY statement returns the following 4 columns:

Syntax Element	Description
`db_name`	The database name
`table_name`	The table name
`partition_name`	The partition name
`healthy`	The health state of tables

Metadata of columns

You can use the SHOW STATS_HISTOGRAMS statement to view the number of different values and the number of NULL in all the columns.

Syntax as follows:

SHOW STATS_HISTOGRAMS [ShowLikeOrWhere]

This statement returns the number of different values and the number of NULL in all the columns. You can use ShowLikeOrWhere to filter the information you need.

Currently, the SHOW STATS_HISTOGRAMS statement returns the following 10 columns:

Syntax Element	Description
`db_name`	The database name
`table_name`	The table name
`partition_name`	The partition name
`column_name`	The column name (when `is_index` is `0`) or the index name (when `is_index` is `1`)
`is_index`	Whether it is an index column or not
`update_time`	The time of the update
`distinct_count`	The number of different values
`null_count`	The number of `NULL`
`avg_col_size`	The average length of columns
correlation	The Pearson correlation coefficient of the column and the integer primary key, which indicates the degree of association between the two columns

Buckets of histogram

You can use the SHOW STATS_BUCKETS statement to view each bucket of the histogram.

The syntax is as follows:

SHOW STATS_BUCKETS [ShowLikeOrWhere]

The diagram is as follows:

SHOW STATS_BUCKETS

This statement returns information about all the buckets. You can use ShowLikeOrWhere to filter the information you need.

Currently, the SHOW STATS_BUCKETS statement returns the following 11 columns:

Syntax Element	Description
`db_name`	The database name
`table_name`	The table name
`partition_name`	The partition name
`column_name`	The column name (when `is_index` is `0`) or the index name (when `is_index` is `1`)
`is_index`	Whether it is an index column or not
`bucket_id`	The ID of a bucket
`count`	The number of all the values that falls on the bucket and the previous buckets
`repeats`	The occurrence number of the maximum value
`lower_bound`	The minimum value
`upper_bound`	The maximum value
`ndv`	The number of different values in the bucket. When `tidb_analyze_version` = `1`, `ndv` is always `0`, which has no actual meaning.

Top-N information

You can use the SHOW STATS_TOPN statement to view the Top-N information currently collected by TiDB.

The syntax is as follows:

SHOW STATS_TOPN [ShowLikeOrWhere];

Currently, the SHOW STATS_TOPN statement returns the following 7 columns:

Syntax Element	Description
`db_name`	The database name
`table_name`	The table name
`partition_name`	The partition name
`column_name`	The column name (when `is_index` is `0`) or the index name (when `is_index` is `1`)
`is_index`	Whether it is an index column or not
`value`	The value of this column
`count`	How many times the value appears

Delete statistics

You can run the DROP STATS statement to delete statistics.

Syntax as follows:

DROP STATS TableName

The statement deletes statistics of all the tables in TableName.

Import and export statistics

Export statistics

The interface to export statistics is as follows:

To obtain the JSON format statistics of the ${table_name} table in the ${db_name} database:

http://${tidb-server-ip}:${tidb-server-status-port}/stats/dump/${db_name}/${table_name}

For example:

curl -s http://127.0.0.1:10080/stats/dump/test/t1 -o /tmp/t1.json

To obtain the JSON format statistics of the ${table_name} table in the ${db_name} database at specific time:

http://${tidb-server-ip}:${tidb-server-status-port}/stats/dump/${db_name}/${table_name}/${yyyyMMddHHmmss}

Import statistics

Note

When you start the MySQL client, use the --local-infile=1 option.

Generally, the imported statistics refer to the JSON file obtained using the export interface.

Syntax:

LOAD STATS 'file_name'

file_name is the file name of the statistics to be imported.

Introduction to Statistics

Histogram

Count-Min Sketch

Top-N values

Collect statistics

Manual collection

Full collection

Incremental collection

Automatic update

Control `ANALYZE` concurrency

`tidb_build_stats_concurrency`

`tidb_distsql_scan_concurrency`

`tidb_index_serial_scan_concurrency`

View `ANALYZE` state

View statistics

Metadata of tables

Health state of tables

Metadata of columns

Buckets of histogram

Top-N information

Delete statistics

Import and export statistics

Export statistics

Import statistics

See also

Was this page helpful?

Introduction to Statistics

Histogram

Count-Min Sketch

Top-N values

Collect statistics

Manual collection

Full collection

Incremental collection

Automatic update

Control ANALYZE concurrency

tidb_build_stats_concurrency

tidb_distsql_scan_concurrency

tidb_index_serial_scan_concurrency

View ANALYZE state

View statistics

Metadata of tables

Health state of tables

Metadata of columns

Buckets of histogram

Top-N information

Delete statistics

Import and export statistics

Export statistics

Import statistics

See also

Was this page helpful?

Control `ANALYZE` concurrency

`tidb_build_stats_concurrency`

`tidb_distsql_scan_concurrency`

`tidb_index_serial_scan_concurrency`

View `ANALYZE` state