Tiered Storage Overview

Tiered storage helps you move infrequently accessed table or partition data to a lower-cost storage tier while keeping SQL semantics unchanged. This page explains the architecture, trade-offs, and scenario guidelines to help you decide when to use Infrequent Access (IA) storage.

Introduction

Tiered storage is a storage tiering capability at the table and partition levels available in TiDB Cloud Premium and TiDB Cloud BYOC. It is designed for infrequently accessed data. You can assign a table or partition to the Infrequent Access (IA) storage class. The system stores the complete dataset in remote object storage (such as Amazon S3 or Alibaba Cloud OSS), while keeping only metadata and on-demand cached hot data segments on local storage.

From the application’s perspective, an IA table behaves like a standard table. All query, transaction, backup, and recovery semantics remain unchanged. The primary difference is in the cost-performance tradeoff: local storage usage is significantly reduced, but cold reads (when the required data is not available in the local cache) incur higher latency because data must first be fetched from remote object storage.

Key features:

• Transparent semantics: All SQL operations, including SELECT, INSERT, UPDATE, and DELETE, behave the same as on standard tables.
• Lower storage cost: Data is stored in low-cost object storage, while only hot data is cached locally. In typical scenarios, storage costs can be reduced by approximately 50%. The exact reduction depends on the IA cache level that you select. For details, see TiDB Cloud Billing.
• Fine-grained control: Supports both table-level and partition-level storage tiering. Partition-level settings take precedence over table-level settings.
• Flexible conversion: Supports bidirectional conversion between IA and standard storage classes without data loss.
• Seamless integration: Fully integrated with Raft Regions, MVCC, BR backup and restore, TiCDC, and other TiDB components.
• Observable and tunable: Conversion progress, IA read metrics, and the cluster-level cache hit rate are visible in SQL outputs and the Cloud Console, and the IA cache level is configurable.

Usage scenarios

This section describes the recommended and not recommended scenarios for using IA storage, as well as a checklist to help you decide whether to use IA.

Recommended scenarios

General rule of thumb: Large data volume + decreasing access frequency over time + occasional queries with no strict response time requirement → suitable for IA.

Not recommended scenarios

• Hot OLTP tables with strict latency sensitivity (core online transaction tables sensitive to every millisecond)
• Datasets requiring frequent large-range scans (AP queries that continuously access large amounts of cold data)
• Tables that need frequent switching between IA and Standard
• Data with highly scattered access patterns and almost no locality
• Scenarios where the access pattern does not follow a "decreasing over time" trend

Decision checklist

Before setting IA, verify each item:

• The data access frequency of the table/partition has been confirmed to be declining from a business perspective
• The table can be changed to a partitioned table, because cold/hot data separation is easier to manage with partitioned tables
• For regular table cold/hot separation, hot data accounts for less than 10% of the table
• Cold data access frequency is very low; for example, query QPS does not exceed 10, to avoid saturating object storage bandwidth
• No need for frequent large-range AP scans on IA tables
• Cold-read latency is acceptable: a SQL execution may issue multiple remote requests, and latency varies with cache state, request parallelism, and the amount of cold data accessed
• Awareness that cold reads have read amplification: a single 100-byte record can exhibit up to 30,000× amplification (approximately 3 MiB of cold data)
• Single query involves no more than 100 MiB of cold data
• Single query accesses no more than 100 rows of cold data
• An observation period has been planned (at least one full business day for the first partition)
• Awareness that switching back IA → Standard takes a long time with significant bandwidth cost

Implementation principles

This section describes the architecture, data storage hierarchy, read amplification analysis, and LSM-Tree write path of tiered storage.

Architecture overview

Tiered storage implementation spans the TiDB → TiKV → Object Store three-layer architecture:

┌──────────────────────────────────────────────────────────────────┐
│ TiDB Schema Layer                                                │
│ · STORAGE_CLASS / ENGINE_ATTRIBUTE written to TiDB schema        │
└──────────────────────────────────────────────────────────────────┘
                              ↓
┌──────────────────────────────────────────────────────────────────┐
│ TiKV Region Management Layer                                     │
│ · IA tables/partitions occupy dedicated regions, avoiding        │
│   hot/cold data in the same shard                                │
└──────────────────────────────────────────────────────────────────┘
                              ↓
┌──────────────────────────────────────────────────────────────────┐
│ TiKV IA Cache Management Layer (IaManager)                       │
│ · Local cache for cold storage data                              │
└──────────────────────────────────────────────────────────────────┘
                              ↓
┌──────────────────────────────────────────────────────────────────┐
│ Remote Object Storage (S3/OSS)                                   │
│ · Full SST data stored as whole objects, not by segment          │
└──────────────────────────────────────────────────────────────────┘

Raft ChangeSet ensures consistency: Storage class changes are replicated to all replicas via Raft ChangeSet. This ensures that every related shard maintains a consistent storage class state across restarts, recovery, splits, and merges, preventing data loss or corruption due to region topology changes.

Local cache and cache level

The IaManager layer keeps a local cache of IA data on each TiKV node. How much data can be cached is determined by the IA cache level, capped by the local disk capacity reserved for the cache.

The cache level is a cluster-wide setting with four options: Economy, Default, Balanced, and Deep. A higher level caches a larger proportion of IA data, which raises the cache hit rate and the cost. Eviction is managed by the system: you cannot specify which data stays in the cache, and you cannot set a different level for an individual table or partition. For how to change the cache level, see Configure and Manage Tiered Storage.

Data storage hierarchy

The internal hierarchy of an SSTable from top to bottom:

SSTable ──→ Segment ──→ Block ──→ KV Pair
(file)      (1 MiB)     (32 KiB)    (single record)

This means a cache miss does not fetch just a single KV record — it loads an entire Segment to local storage. If subsequent queries hit data within the same segment, they benefit from hot-read performance. However, for one-time queries with no subsequent access, the read amplification penalty is relatively high.

Segments organize only the local cache. In object storage, an SST file is stored as a single whole object and is not split into segments, so changing the segment size does not rewrite any data in object storage. On TiDB Cloud BYOC, the segment size is configurable. For details, see Configure and Manage Tiered Storage.

Read amplification analysis

Read amplification path on a cache miss:

User queries 1 record (100 Bytes)
→ Block cache miss
→ TiKV loads segments from 3 LSM levels from object storage
→ Approximately 3 MiB data fetched from object storage to local
→ Read amplification: ~30,000×

The volume fetched per cache miss scales with the segment size. The figures above assume the default segment size of 1 MiB.

This amplification manifests differently in two scenarios:

• With subsequent reuse: The loaded segment stays in the IA cache; subsequent hits become hot reads, amortizing the initial amplification cost
• One-time query: e.g., an ad-hoc analysis running a large-range scan — the loaded data is never reused, making the read cost very high. Additionally, newly loaded data evicts "old data" from the local cache — this old data might be real hot spots, and their eviction can trigger new cache misses, creating a cascading performance impact

Therefore, tiered storage is best suited for small, concentrated query patterns rather than frequent large-range scans.

LSM-Tree write path

The write path remains the same as Standard tables:

INSERT/UPDATE/DELETE
→ Memtable (hot write, unaffected by IA)
→ L0 SST (hot write, unaffected by IA)
→ L1+ SST (after compaction, opened in IA mode based on storage class)

• Writes still enter memtable/L0 first — the local hot path does not directly become a remote write
• L1+ files matching IA conditions are opened in IA mode after reload/compaction
• Hot write paths like memtable and L0 do not directly enter IA
• The primary IA target is the Write CF L1+ layer

This also explains why the storage cost reduction is approximate rather than exact. IA storage space covers only the L1 and deeper layers of a table. The memtable and L0 files remain on local disk and are not counted as IA storage, so the total space of a table is the sum of its memtable, L0, and L1+ data. How much of a table has actually moved to IA storage depends on the write rate and on compaction progress.

Conversion efficiency

An ALTER TABLE statement that changes the storage class, in either the STORAGE_CLASS or the ENGINE_ATTRIBUTE form, completes within seconds because it only updates the schema metadata. The region-level data migration then runs asynchronously in TiKV, so the total duration depends on the data volume and the conversion direction.

Standard → IA

Test reference: Approximately 1 TB of logical data (including indexes) completed conversion within 5 minutes, with negligible impact on QPS and latency during the process. Single TiKV CPU increased by about 0.5c and recovered within approximately 5 minutes.

TiDB schema takes effect → Schema Manager sync (30s) → TiKV broadcast
→ Region alignment / Split → ChangeSet updates Shard → Reload files

IA → Standard

Test reference: 2.09 TB of logical data (including indexes) took approximately 3 hours 10 minutes (~1.61k regions/hour per TiKV), object storage GET throughput was approximately 1.6 GiB/s. During conversion, Standard partition QPS dropped by about 3.78%, P99 increased by about 18.63%, and single TiKV CPU increased by about 0.5c.

Same chain as above + full object storage data download to local

To track a conversion in progress with SHOW STORAGE_CLASS TRANSITIONS, or to query the duration of past conversions in mysql.tidb_storage_class_transition_history, see Tiered Storage Observability.