Doc Menu

Configure a TiDB Cluster in Kubernetes

This document introduces how to configure a TiDB cluster for production deployment. It covers the following content:

Configure resources

Before deploying a TiDB cluster, it is necessary to configure the resources for each component of the cluster depending on your needs. PD, TiKV, and TiDB are the core service components of a TiDB cluster. In a production environment, you need to configure resources of these components according to their needs. For details, refer to Hardware Recommendations.

To ensure the proper scheduling and stable operation of the components of the TiDB cluster in Kubernetes, it is recommended to set Guaranteed-level quality of service (QoS) by making limits equal to requests when configuring resources. For details, refer to Configure Quality of Service for Pods.

If you are using a NUMA-based CPU, you need to enable Static's CPU management policy on the node for better performance. In order to allow the TiDB cluster component to monopolize the corresponding CPU resources, the CPU quota must be an integer greater than or equal to 1, apart from setting Guaranteed-level QoS as mentioned above. For details, refer to Control CPU Management Policies on the Node.

Configure TiDB deployment

To configure a TiDB deployment, you need to configure the TiDBCluster CR. Refer to the TidbCluster example for an example. For the complete configurations of TiDBCluster CR, refer to API documentation.

Note:

It is recommended to organize configurations for a TiDB cluster under a directory of cluster_name and save it as ${cluster_name}/tidb-cluster.yaml. The modified configuration is not automatically applied to the TiDB cluster by default. The new configuration file is loaded only when the Pod restarts.

Cluster name

The cluster name can be configured by changing metadata.name in the TiDBCuster CR.

Version

Usually, components in a cluster are in the same version. It is recommended to configure spec.<pd/tidb/tikv/pump/tiflash/ticdc>.baseImage and spec.version, if you need to configure different versions for different components, you can configure spec.<pd/tidb/tikv/pump/tiflash/ticdc>.version. Here are the formats of the parameters:

  • spec.version: the format is imageTag, such as v4.0.8

  • spec.<pd/tidb/tikv/pump/tiflash/ticdc>.baseImage: the format is imageName, such as pingcap/tidb

  • spec.<pd/tidb/tikv/pump/tiflash/ticdc>.version: the format is imageTag, such as v4.0.8

configUpdateStrategy

It is recommended that you configure spec.configUpdateStrategy: RollingUpdate to enable automatic update of configurations. This way, every time the configuration is updated, all components are rolling updated automatically, and the modified configuration is applied to the cluster.

enableDynamicConfiguration

It is recommended that you configure spec.enableDynamicConfiguration: true to enable the dynamic configuration feature.

Versions required:

  • TiDB 4.0.1 or later versions
  • TiDB Operator 1.1.1 or later versions

pvReclaimPolicy

It is recommended that you configure spec.pvReclaimPolicy: Retain to ensure that the PV is retained even if the PVC is deleted. This is to ensure your data safety.

Storage

You can set the storage class by modifying storageClassName of each component in ${cluster_name}/tidb-cluster.yaml and ${cluster_name}/tidb-monitor.yaml. For the storage classes supported by the Kubernetes cluster, check with your system administrator.

Different components of a TiDB cluster have different disk requirements. Before deploying a TiDB cluster, select the appropriate storage class for each component according to the storage classes supported by the current Kubernetes cluster and usage scenario.

For the production environment, local storage is recommended for TiKV. The actual local storage in Kubernetes clusters might be sorted by disk types, such as nvme-disks and sas-disks.

For the demonstration environment or functional verification, you can use network storage, such as ebs and nfs.

Note:

When you create the TiDB cluster, if you set a storage class that does not exist in the Kubernetes cluster, then the TiDB cluster creation goes to the Pending state. In this situation, you must destroy the TiDB cluster in Kubernetes and retry the creation.

Multiple disks mounting

TiDB Operator supports mounting multiple PVs for PD, TiDB, and TiKV, which can be used for data writing for different purposes.

You can configure the storageVolumes field for each component to describe multiple user-customized PVs.

The meanings of the related fields are as follows:

  • storageVolume.name: The name of the PV.
  • storageVolume.storageClassName: The StorageClass that the PV uses. If not configured, spec.pd/tidb/tikv.storageClassName will be used.
  • storageVolume.storageSize: The storage size of the requested PV.
  • storageVolume.mountPath: The path of the container to mount the PV to.

For example:

  pd:
    baseImage: pingcap/pd
    replicas: 1
    # if storageClassName is not set, the default Storage Class of the Kubernetes cluster will be used
    # storageClassName: local-storage
    requests:
      storage: "1Gi"
    config:
      log:
        file:
          name: /var/log/pd
        level: "warn"
    storageVolumes:
      - name: log
        storageSize: "2Gi"
        mountPath: "/var/log/pd"
  tidb:
    baseImage: pingcap/tidb
    replicas: 1
    service:
      type: ClusterIP
    config:
      log:
        file:
          name: /var/log/tidb
        level: "warn"
    storageVolumes:
      - name: log
        storageSize: "2Gi"
        mountPath: "/var/log/tidb"
  tikv:
    baseImage: pingcap/tikv
    replicas: 1
    # if storageClassName is not set, the default Storage Class of the Kubernetes cluster will be used
    # storageClassName: local-storage
    requests:
      storage: "1Gi"
    config:
      storage:
        # In basic examples, you can set this to avoid using too much storage.
        reserve-space: "0MB"
      rocksdb:
        wal-dir: "/data_sbi/tikv/wal"
      titan:
        dirname: "/data_sbj/titan/data"
    storageVolumes:
      - name: wal
        storageSize: "2Gi"
        mountPath: "/data_sbi/tikv/wal"
      - name: titan
        storageSize: "2Gi"
        mountPath: "/data_sbj/titan/data"

mountClusterClientSecret

It is recommended that you configure spec.pd.mountClusterClientSecret: true and spec.tikv.mountClusterClientSecret: true, so that TiDB Operator can mount the ${cluster_name}-cluster-client-secret certificates to the PD and TiKV containers automatically, which can make the use of pd-ctl and tikv-ctl easier.

Cluster topology

PD/TiKV/TiDB

The deployed cluster topology by default has three PD Pods, three TiKV Pods, and two TiDB Pods. In this deployment topology, the scheduler extender of TiDB Operator requires at least three nodes in the Kubernetes cluster to provide high availability. You can modify the replicas configuration to change the number of pods for each component.

Note:

If the number of Kubernetes cluster nodes is less than three, one PD Pod goes to the Pending state, and neither TiKV Pods nor TiDB Pods are created. When the number of nodes in the Kubernetes cluster is less than three, to start the TiDB cluster, you can reduce both the number of PD Pods and the number of TiKV Pods in the default deployment to 1.

Enable TiFlash

If you want to enable TiFlash in the cluster, configure spec.pd.config.replication.enable-placement-rules to true and configure spec.tiflash in the ${cluster_name}/tidb-cluster.yaml file as follows:

  pd:
    config:
      ...
      replication:
        enable-placement-rules: "true"
        ...
  tiflash:
    baseImage: pingcap/tiflash
    maxFailoverCount: 3
    replicas: 1
    storageClaims:
    - resources:
        requests:
          storage: 100Gi
      storageClassName: local-storage

TiFlash supports mounting multiple Persistent Volumes (PVs). If you want to configure multiple PVs for TiFlash, configure multiple resources in tiflash.storageClaims, each resources with a separate storage request and storageClassName. For example:

  tiflash:
    baseImage: pingcap/tiflash
    maxFailoverCount: 3
    replicas: 1
    storageClaims:
    - resources:
        requests:
          storage: 100Gi
      storageClassName: local-storage
    - resources:
        requests:
          storage: 100Gi
      storageClassName: local-storage

TiFlash mounts all PVs to directories such as /data0 and /data1 in the container in the order of configuration. TiFlash has four log files. The proxy log is printed in the standard output of the container. The other three logs are stored in the disk under the /data0 directory by default, which are /data0/logs/flash_cluster_manager.log, / data0/logs/error.log, /data0/logs/server.log. To modify the log storage path, refer to Configure TiFlash parameters.

Warning:

Since TiDB Operator will mount PVs automatically in the order of the items in the storageClaims list, if you need to add more disks to TiFlash, make sure to append the new item only to the end of the original items, and DO NOT modify the order of the original items.

Enable TiCDC

If you want to enable TiCDC in the cluster, you can add TiCDC spec to the TiDBCluster CR. For example:

  spec:
    ticdc:
      baseImage: pingcap/ticdc
      replicas: 3

Deploy Enterprise Edition

To deploy Enterprise Edition of TiDB/PD/TiKV/TiFlash/TiCDC, edit the db.yaml file to set spec.<tidb/pd/tikv/tiflash/ticdc>.baseImage to the enterprise image (pingcap/<tidb/pd/tikv/tiflash/ticdc>-enterprise).

For example:

spec:
  ...
  pd:
    baseImage: pingcap/pd-enterprise
  ...
  tikv:
    baseImage: pingcap/tikv-enterprise

Configure TiDB components

This section introduces how to configure the parameters of TiDB/TiKV/PD/TiFlash/TiCDC.

The current TiDB Operator v1.1 supports all parameters of TiDB v4.0.

Configure TiDB parameters

TiDB parameters can be configured by spec.tidb.config in TidbCluster Custom Resource.

For example:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: basic
spec:
....
  tidb:
    image: pingcap/tidb:v4.0.8
    imagePullPolicy: IfNotPresent
    replicas: 1
    service:
      type: ClusterIP
    config:
      split-table: true
      oom-action: "log"
    requests:
      cpu: 1

Since v1.1.6, TiDB Operator supports passing raw TOML configuration to the component:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: basic
spec:
....
  tidb:
    image: pingcap/tidb:v4.0.8
    imagePullPolicy: IfNotPresent
    replicas: 1
    service:
      type: ClusterIP
    config: |
      split-table = true
      oom-action = "log"
    requests:
      cpu: 1

For all the configurable parameters of TiDB, refer to TiDB Configuration File.

Note:

If you deploy your TiDB cluster using CR, make sure that Config: {} is set, no matter you want to modify config or not. Otherwise, TiDB components might not be started successfully. This step is meant to be compatible with Helm deployment.

Configure TiKV parameters

TiKV parameters can be configured by spec.tikv.config in TidbCluster Custom Resource.

For example:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: basic
spec:
....
  tikv:
    image: pingcap/tikv:v4.0.8
    config:
      log-level: "info"
      slow-log-threshold: "1s"
    replicas: 1
    requests:
      cpu: 2

Since v1.1.6, TiDB Operator supports passing raw TOML configuration to the component:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: basic
spec:
....
  tikv:
    image: pingcap/tikv:v4.0.8
    config: |
      #  [storage]
      #    reserve-space = "2MB"
    replicas: 1
    requests:
      cpu: 2

For all the configurable parameters of TiKV, refer to TiKV Configuration File.

Note:

If you deploy your TiDB cluster using CR, make sure that Config: {} is set, no matter you want to modify config or not. Otherwise, TiKV components might not be started successfully. This step is meant to be compatible with Helm deployment.

Configure PD parameters

PD parameters can be configured by spec.pd.config in TidbCluster Custom Resource.

For example:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: basic
spec:
.....
  pd:
    image: pingcap/pd:v4.0.8
    config:
      lease: 3
      enable-prevote: true

Since v1.1.6, TiDB Operator supports passing raw TOML configuration to the component:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: basic
spec:
.....
  pd:
    image: pingcap/pd:v4.0.8
    config: |
      lease = 3
      enable-prevote = true

For all the configurable parameters of PD, refer to PD Configuration File.

Note:

If you deploy your TiDB cluster using CR, make sure that Config: {} is set, no matter you want to modify config or not. Otherwise, PD components might not be started successfully. This step is meant to be compatible with Helm deployment.

Configure TiFlash parameters

TiFlash parameters can be configured by spec.tiflash.config in TidbCluster Custom Resource.

For example:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: basic
spec:
  ...
  tiflash:
    config:
      config:
       flash:
          flash_cluster:
            log: "/data0/logs/flash_cluster_manager.log"
        logger:
          count: 10
          level: information
          errorlog: "/data0/logs/error.log"
          log: "/data0/logs/server.log"

Since v1.1.6, TiDB Operator supports passing raw TOML configuration to the component:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: basic
spec:
  ...
  tiflash:
    config:
      config: |
        [flash]
          [flash.flash_cluster]
            log = "/data0/logs/flash_cluster_manager.log"
        [logger]
          count = 10
          level = "information"
          errorlog = "/data0/logs/error.log"
          log = "/data0/logs/server.log"

For all the configurable parameters of TiFlash, refer to TiFlash Configuration File.

Configure TiCDC start parameters

You can configure TiCDC start parameters through spec.ticdc.config in TidbCluster Custom Resource.

For example:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: basic
spec:
  ...
  ticdc:
    config:
      timezone: UTC
      gcTTL: 86400
      logLevel: info

For all configurable start parameters of TiCDC, see TiCDC configuration.

Configure high availability

Note:

TiDB Operator provides a custom scheduler that guarantees TiDB service can tolerate host-level failures through the specified scheduling algorithm. Currently, the TiDB cluster uses this scheduler as the default scheduler, which is configured through the item spec.schedulerName. This section focuses on configuring a TiDB cluster to tolerate failures at other levels such as rack, zone, or region. This section is optional.

TiDB is a distributed database and its high availability must ensure that when any physical topology node fails, not only the service is unaffected, but also the data is complete and available. The two configurations of high availability are described separately as follows.

High availability of TiDB service

High availability at other levels (such as rack, zone, region) is guaranteed by Affinity's PodAntiAffinity. PodAntiAffinity can avoid the situation where different instances of the same component are deployed on the same physical topology node. In this way, disaster recovery is achieved. Detailed user guide for Affinity: Affinity & AntiAffinity.

The following is an example of a typical service high availability setup:

affinity:
 podAntiAffinity:
   preferredDuringSchedulingIgnoredDuringExecution:
   # this term works when the nodes have the label named region
   - weight: 10
     podAffinityTerm:
       labelSelector:
         matchLabels:
           app.kubernetes.io/instance: ${cluster_name}
           app.kubernetes.io/component: "pd"
       topologyKey: "region"
       namespaces:
       - ${namespace}
   # this term works when the nodes have the label named zone
   - weight: 20
     podAffinityTerm:
       labelSelector:
         matchLabels:
           app.kubernetes.io/instance: ${cluster_name}
           app.kubernetes.io/component: "pd"
       topologyKey: "zone"
       namespaces:
       - ${namespace}
   # this term works when the nodes have the label named rack
   - weight: 40
     podAffinityTerm:
       labelSelector:
         matchLabels:
           app.kubernetes.io/instance: ${cluster_name}
           app.kubernetes.io/component: "pd"
       topologyKey: "rack"
       namespaces:
       - ${namespace}
   # this term works when the nodes have the label named kubernetes.io/hostname
   - weight: 80
     podAffinityTerm:
       labelSelector:
         matchLabels:
           app.kubernetes.io/instance: ${cluster_name}
           app.kubernetes.io/component: "pd"
       topologyKey: "kubernetes.io/hostname"
       namespaces:
       - ${namespace}

High availability of data

Before configuring the high availability of data, read Information Configuration of the Cluster Typology which describes how high availability of TiDB cluster is implemented.

To add the data high availability feature in Kubernetes:

  1. Set the label collection of topological location for PD

    Replace the location-labels information in the pd.config with the label collection that describes the topological location on the nodes in the Kubernetes cluster.

    Note:

    • For PD versions < v3.0.9, the / in the label name is not supported.
    • If you configure host in the location-labels, TiDB Operator will get the value from the kubernetes.io/hostname in the node label.
  2. Set the topological information of the Node where the TiKV node is located.

    TiDB Operator automatically obtains the topological information of the Node for TiKV and calls the PD interface to set this information as the information of TiKV's store labels. Based on this topological information, the TiDB cluster schedules the replicas of the data.

    If the Node of the current Kubernetes cluster does not have a label indicating the topological location, or if the existing label name of topology contains /, you can manually add a label to the Node by running the following command:

    kubectl label node ${node_name} region=${region_name} zone=${zone_name} rack=${rack_name} kubernetes.io/hostname=${host_name}

    In the command above, region, zone, rack, and kubernetes.io/hostname are just examples. The name and number of the label to be added can be arbitrarily defined, as long as it conforms to the specification and is consistent with the labels set by location-labels in pd.config.