- Key Features
- Horizontal Scalability
- MySQL Compatible Syntax
- Replicate from and to MySQL
- Distributed Transactions with Strong Consistency
- Cloud Native Architecture
- Minimize ETL with HTAP
- Fault Tolerance & Recovery with Raft
- Automatic Rebalancing
- Deployment and Orchestration with Ansible, Kubernetes, Docker
- JSON Support
- Spark Integration
- Read Historical Data Without Restoring from Backup
- Fast Import and Restore of Data
- Hybrid of Column and Row Storage
- SQL Plan Management
- Open Source
- Online Schema Changes
- Key Features
- Get Started
- From Binary Tarball
- Orchestrated Deployment
- Geographic Redundancy
- SQL Language Structure
- Data Types
- Numeric Types
- Date and Time Types
- String Types
- Functions and Operators
- Function and Operator Reference
- Type Conversion in Expression Evaluation
- Control Flow Functions
- String Functions
- Numeric Functions and Operators
- Date and Time Functions
- Bit Functions and Operators
- Cast Functions and Operators
- Encryption and Compression Functions
- Information Functions
- JSON Functions
- Aggregate (GROUP BY) Functions
- Miscellaneous Functions
- Precision Math
- SQL Statements
ADMIN CANCEL DDL
ADMIN CHECKSUM TABLE
ADMIN CHECK [TABLE|INDEX]
ADMIN SHOW DDL [JOBS|QUERIES]
CREATE TABLE LIKE
SET [NAMES|CHARACTER SET]
SET [GLOBAL|SESSION] <variable>
SHOW CHARACTER SET
SHOW [FULL] COLUMNS FROM
SHOW CREATE TABLE
SHOW [FULL] FIELDS FROM
SHOW INDEXES [FROM|IN]
SHOW INDEX [FROM|IN]
SHOW KEYS [FROM|IN]
SHOW [FULL] PROCESSSLIST
SHOW [FULL] TABLES
SHOW TABLE STATUS
SHOW [GLOBAL|SESSION] VARIABLES
- System Databases
- Key Monitoring Metrics
- Best Practices
- TiDB Binlog
- TiDB Lightning
- All Releases
This page explains the special terms used in TiDB Lightning's logs, monitoring, configurations, and documentation.
Because TiDB Lightning imports data without going through TiDB, the statistics information is not automatically updated. Therefore, TiDB Lightning explicitly analyzes every table after importing. This step can be omitted by setting the
post-restore.analyze configuration to
Every table has an associated
AUTO_INCREMENT_ID counter to provide the default value of an auto-incrementing column. In TiDB, this counter is additionally used to assign row IDs.
Because TiDB Lightning imports data without going through TiDB, the
AUTO_INCREMENT_ID counter is not automatically updated. Therefore, TiDB Lightning explicitly alters
AUTO_INCREMENT_ID to a valid value. This step is always performed, even if the table has no
A configuration list that specifies which tables to be imported and which should be excluded.
See TiDB Lightning Table Filter for details.
TiDB Lightning continuously saves its progress into a local file or a remote database while importing. This allows it to resume from an intermediate state should it crashes in the process. See the Checkpoints section for details.
In TiDB Lightning, the checksum of a table is a set of 3 numbers calculated from the content of each KV pair in that table. These numbers are respectively:
- the number of KV pairs,
- total length of all KV pairs, and
- the bitwise-XOR of CRC-64-ECMA value each pair.
TiDB Lightning validates the imported data by comparing the local and remote checksums of every table. The program would stop if any pair does not match. You can skip this check by setting the
post-restore.checksum configuration to
See also the Troubleshooting guide for how to properly handle checksum mismatch.
Equivalent to a single file in the data source.
An operation that merges multiple small SST files into one large SST file, and cleans up the deleted entries. TiKV automatically compacts data in background while TiDB Lightning is importing.
For legacy reasons, you can still configure TiDB Lightning to explicitly trigger a compaction every time a table is imported. However, this is not recommended and the corresponding settings are disabled by default.
See RocksDB's wiki page on Compaction for its technical details.
An engine for sorting actual row data.
When a table is very large, its data is placed into multiple data engines to improve task pipelining and save space of TiKV Importer. By default, a new data engine is opened for every 100 GB of SQL data, which can be configured through the
TiDB Lightning processes multiple data engines concurrently. This is controlled by the
In TiKV Importer, an engine is a RocksDB instance for sorting KV pairs.
TiDB Lightning transfers data to TiKV Importer through engines. It first opens an engine, sends KV pairs to it (with no particular order), and finally closes the engine. The engine sorts the received KV pairs after it is closed. These closed engines can then be further uploaded to the TiKV stores for ingestion.
Engines use TiKV Importer's
import-dir as temporary storage, which are sometimes referred to as "engine files".
A configuration that optimizes TiKV for writing at the cost of degraded read speed and space usage.
An engine for sorting indices.
Regardless of number of indices, every table is associated with exactly one index engine.
TiDB Lightning processes multiple index engines concurrently. This is controlled by the
lightning.index-concurrency setting. Since every table has exactly one index engine, this also configures the maximum number of tables to process at the same time.
An operation which inserts the entire content of an SST file into the RocksDB (TiKV) store.
Ingestion is a very fast operation compared with inserting KV pairs one by one. This operation is the determinant factor for the performance of TiDB Lightning.
See RocksDB's wiki page on Creating and Ingesting SST files for its technical details.
Abbreviation of "key-value pair".
A routine which parses SQL or CSV rows to KV pairs. Multiple KV encoders run in parallel to speed up processing.
The checksum of a table calculated by TiDB Lightning itself before sending the KV pairs to TiKV Importer.
The mode where import mode is disabled.
The checksum of a table calculated by TiDB after it has been imported.
An operation that randomly reassigns the leader and the peers of a Region. Scattering ensures that the imported data are distributed evenly among TiKV stores. This reduces stress on PD.
An engine is typically very large (around 100 GB), which is not friendly to TiKV if treated as a single region. TiKV Importer splits an engine into multiple small SST files (configurable by TiKV Importer's
import.region-split-size setting) before uploading.
SST is the abbreviation of "sorted string table". An SST file is RocksDB's (and thus TiKV's) native storage format of a collection of KV pairs.