📣

TiDB Cloud Premium is now in public preview. Unlimited growth, instant elasticity, advanced security for enterprise workloads. Try it out →

Apache Iceberg™ Tables

TiDB Cloud Lake can connect to Apache Iceberg™ catalogs so that you can query Iceberg tables without loading their data into Fuse tables. You can also create and write Iceberg tables when the connected catalog supports write operations.

When to Use Iceberg

Use Iceberg when:

Your data is already managed by an Iceberg catalog.
Multiple query engines need to share the same table metadata and object storage.
You need Iceberg capabilities such as schema evolution and snapshots.
You want to query or write Iceberg tables from TiDB Cloud Lake.

Create an Iceberg Catalog

Create a catalog before accessing Iceberg databases and tables.

Syntax

CREATE CATALOG <catalog_name>
TYPE = ICEBERG
CONNECTION = (
    TYPE = '<catalog_type>'
    [ ADDRESS = '<catalog_address>' ]
    [ WAREHOUSE = '<warehouse_location>' ]
    [ "<connection_parameter>" = '<connection_parameter_value>' ]
    ...
);

Parameters

Parameter	Required?	Description
`<catalog_name>`	Yes	Name of the catalog in TiDB Cloud Lake.
`TYPE`	Yes	Catalog engine. Set this value to `ICEBERG`.
`CONNECTION`	Yes	Connection properties for the Iceberg catalog and its storage.
`TYPE` inside `CONNECTION`	Yes	Iceberg catalog type: `rest`, `glue`, `storage`, or `hive`.
`ADDRESS`	Depends on catalog type	Catalog service endpoint or Hive Metastore address.
`WAREHOUSE`	Depends on catalog type	Warehouse location used by the catalog.
`<connection_parameter>`	Depends on catalog type	Catalog, authentication, and object storage properties.

The following connection parameters are available for S3-compatible storage:

Connection Parameter	Description
`s3.endpoint`	S3-compatible service endpoint.
`s3.access-key-id`	S3 access key ID.
`s3.secret-access-key`	S3 secret access key.
`s3.session-token`	Session token used with temporary credentials.
`s3.region`	S3 region.
`client.region`	Region used by the client. This value takes precedence over `s3.region`.
`s3.path-style-access`	Whether to use path-style S3 access.
`s3.sse.type`	Server-side encryption type.
`s3.sse.key`	KMS key ID or customer-provided encryption key.
`s3.sse.md5`	MD5 checksum for a customer-provided encryption key.
`client.assume-role.arn`	ARN of the IAM role to assume.
`client.assume-role.external-id`	External ID used when assuming an IAM role.
`client.assume-role.session-name`	Session name used when assuming an IAM role.
`s3.allow-anonymous`	Whether to allow anonymous access to public storage.
`s3.disable-ec2-metadata`	Whether to disable credentials from EC2 instance metadata.
`s3.disable-config-load`	Whether to disable credentials and settings from local configuration sources.

Supported Catalog Types

TiDB Cloud Lake supports the following Iceberg catalog types:

Catalog Type	`TYPE` Value	Connection Requirements
REST	`rest`	REST catalog address, warehouse location, and storage properties.
AWS Glue	`glue`	Glue region and authentication properties, plus S3 storage properties.
Storage (Amazon S3 Tables)	`storage`	Table bucket ARN and AWS client authentication properties.
Hive Metastore	`hive`	Hive Metastore address, warehouse location, and storage properties.

The Storage catalog supports the following AWS client properties:

Connection Parameter	Description
`table_bucket_arn`	ARN of the Amazon S3 Tables table bucket.
`profile_name`	AWS profile name.
`region_name`	AWS region.
`aws_access_key_id`	AWS access key ID.
`aws_secret_access_key`	AWS secret access key.
`aws_session_token`	AWS session token used with temporary credentials.

Manage and Query Iceberg Catalogs

Use the following statements to inspect and select catalogs:

SHOW CREATE CATALOG <catalog_name>;

SHOW CATALOGS [ LIKE '<pattern>' | WHERE <expression> ];

USE CATALOG <catalog_name>;

For more information, see SHOW CREATE CATALOG and SHOW CATALOGS.

After selecting a catalog, use standard SQL to query its tables:

SELECT <select_list>
FROM [ <catalog_name>. ]<database_name>.<table_name>
[ WHERE <condition> ];

Data Type Mapping

The following table shows the supported mappings from Iceberg types to TiDB Cloud Lake types. Iceberg types not listed here are not supported.

Apache Iceberg™	TiDB Cloud Lake
BOOLEAN	BOOLEAN
INT	INT32
LONG	INT64
DATE	DATE
TIMESTAMP / TIMESTAMPZ	TIMESTAMP
FLOAT	FLOAT
DOUBLE	DOUBLE
STRING / BINARY	STRING
DECIMAL	DECIMAL
LIST	ARRAY
MAP	MAP
STRUCT	TUPLE

Refresh Cached Metadata

TiDB Cloud Lake caches Iceberg catalog metadata after the first query. The metadata cache is valid for 10 minutes by default and is refreshed asynchronously.

Use the following statements when you need to refresh cached metadata immediately:

USE CATALOG <catalog_name>;
ALTER DATABASE <database_name> REFRESH CACHE;
ALTER TABLE <database_name>.<table_name> REFRESH CACHE;

TiDB Cloud Lake also supports caching table data read from Iceberg catalogs.

Write to Iceberg Tables

You can create and write Iceberg tables in a catalog that supports write operations.

Create a Table

CREATE TABLE [ <database_name>. ]<table_name> (
    <column_name> <data_type> [ , ... ]
)
ENGINE = ICEBERG
[ PARTITION BY ( <column_name> [ , ... ] ) ];

Parameter	Description
`ENGINE = ICEBERG`	Stores the table in Iceberg format.
`PARTITION BY`	Defines one or more partition columns.

The following TiDB Cloud Lake data types are supported when writing Iceberg tables:

TiDB Cloud Lake Type	Apache Iceberg™ Type
BOOLEAN	Boolean
INT	Int
BIGINT	Long
FLOAT	Float
DOUBLE	Double
STRING	String
DATE	Date
TIMESTAMP	Timestamp

Insert Data

Use INSERT INTO to write rows to an Iceberg table:

INSERT INTO [ <database_name>. ]<table_name>
[ ( <column_name> [ , ... ] ) ]
VALUES ( <value> [ , ... ] ) [ , ... ];

Both partitioned and non-partitioned Iceberg tables support single-row and multi-row inserts. For partitioned tables, TiDB Cloud Lake routes rows to the corresponding partitions.

Iceberg Table Functions

Use the following table functions to inspect Iceberg metadata: