How Fuse Engine Works
Fuse Engine
Fuse Engine is TiDB Cloud Lake's core storage engine, optimized for managing petabyte-scale data efficiently on cloud object storage. By default, tables created in TiDB Cloud Lake automatically use this engine (ENGINE=FUSE). Inspired by Git, its snapshot-based design enables powerful data versioning (like Time Travel) and provides high query performance through advanced pruning and indexing.
This document explains its core concepts and how it works.
Core Concepts
Fuse Engine organizes data using three core structures, mirroring Git:
- Snapshots (Like Git Commits): Immutable references defining the table's state at a point in time by pointing to specific Segments. Enables Time Travel.
- Segments (Like Git Trees): Collections of Blocks with summary statistics used for fast data skipping (pruning). Can be shared across Snapshots.
- Blocks (Like Git Blobs): Immutable data files (Parquet format) holding the actual rows and detailed column-level statistics for fine-grained pruning.
Table HEAD
│
▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ SEGMENT A │◄────│ SNAPSHOT 2 │────►│ SEGMENT B │
│ │ │ Previous: │ │ │
└───────┬───────┘ │ SNAPSHOT 1 │ └───────┬───────┘
│ └───────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────┐ │
│ │ SNAPSHOT 1 │ │
│ │ │ │
│ └───────────────┘ │
│ │
▼ ▼
┌───────────────┐ ┌───────────────┐
│ BLOCK 1 │ │ BLOCK 2 │
│ (cloud.txt) │ │(warehouse.txt)│
└───────────────┘ └───────────────┘
How Writing Works
When you add data to a table, Fuse Engine creates a chain of objects. Let's walk through this process step by step:
Step 1: Create a table
CREATE TABLE git(file VARCHAR, content VARCHAR);
At this point, the table exists but contains no data:
(Empty table with no data)
Step 2: Insert first data
INSERT INTO git VALUES('cloud.txt', '2022/05/06, Datalake, Cloud');
After the first insert, Fuse Engine creates the initial snapshot, segment, and block:
Table HEAD
│
▼
┌───────────────┐
│ SNAPSHOT 1 │
│ │
└───────┬───────┘
│
▼
┌───────────────┐
│ SEGMENT A │
│ │
└───────┬───────┘
│
▼
┌───────────────┐
│ BLOCK 1 │
│ (cloud.txt) │
└───────────────┘
Step 3: Insert more data
INSERT INTO git VALUES('warehouse.txt', '2022/05/07, Datalake, Warehouse');
When we insert more data, Fuse Engine creates a new snapshot that references both the original segment and a new segment:
Table HEAD
│
▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ SEGMENT A │◄────│ SNAPSHOT 2 │────►│ SEGMENT B │
│ │ │ Previous: │ │ │
└───────┬───────┘ │ SNAPSHOT 1 │ └───────┬───────┘
│ └───────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────┐ │
│ │ SNAPSHOT 1 │ │
│ │ │ │
│ └───────────────┘ │
│ │
▼ ▼
┌───────────────┐ ┌───────────────┐
│ BLOCK 1 │ │ BLOCK 2 │
│ (cloud.txt) │ │(warehouse.txt)│
└───────────────┘ └───────────────┘
How Reading Works
When you query data, Fuse Engine uses smart pruning to find your data efficiently:
Query: SELECT * FROM git WHERE file = 'cloud.txt';
Table HEAD
│
▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ SEGMENT A │◄────│ SNAPSHOT 2 │────►│ SEGMENT B │
│ CHECK │ │ │ │ CHECK │
└───────┬───────┘ └───────────────┘ └───────────────┘
│ ✗
│ (Skip - doesn't contain
│ 'cloud.txt')
▼
┌───────────────┐
│ BLOCK 1 │
│ CHECK │
└───────┬───────┘
│
│ ✓ (Contains 'cloud.txt')
▼
Read this block
Smart Pruning Process
┌─────────────────────────────────────────┐
│ Query: WHERE file = 'cloud.txt' │
└─────────────────┬───────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Check SEGMENT A │
│ Min file value: 'cloud.txt' │
│ Max file value: 'cloud.txt' │
│ │
│ Result: ✓ Might contain 'cloud.txt' │
└─────────────────┬───────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Check SEGMENT B │
│ Min file value: 'warehouse.txt' │
│ Max file value: 'warehouse.txt' │
│ │
│ Result: ✗ Cannot contain 'cloud.txt' │
└─────────────────┬───────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Check BLOCK 1 in SEGMENT A │
│ Min file value: 'cloud.txt' │
│ Max file value: 'cloud.txt' │
│ │
│ Result: ✓ Contains 'cloud.txt' │
└─────────────────┬───────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Read only BLOCK 1 │
└─────────────────────────────────────────┘
Snapshot-Based Features
Fuse Engine's snapshot architecture enables powerful data management capabilities:
Time Travel
Query data as it existed at any point in time. Enables data branching, tagging, and governance with complete audit trails and error recovery.
Zero-Copy Schema Evolution
Modify your table's structure (add columns, drop columns, rename, change types) without rewriting any underlying data files.
- Changes are metadata-only operations recorded in new Snapshots.
- This is instantaneous, requires no downtime, and avoids costly data migration tasks. Older data remains accessible with its original schema.
Advanced Indexing for Query Acceleration (Fuse Engine)
Beyond basic block/segment pruning using statistics, Fuse Engine offers specialized secondary indexes to further accelerate specific query patterns:
Comparison: TiDB Cloud Lake Fuse Engine vs. Apache Iceberg
Note: This comparison focuses specifically on table format features. As TiDB Cloud Lake's native table format, Fuse evolves, aiming to improve usability and performance. Features shown are current; expect changes.