Sign InTry Free

Upgrade a TiDB Cluster on Kubernetes

If you deploy and manage your TiDB clusters on Kubernetes using TiDB Operator, you can upgrade your TiDB clusters using the rolling update feature. Rolling update can limit the impact of upgrade on your application.

This document describes how to upgrade a TiDB cluster on Kubernetes using rolling updates.

Rolling update introduction

Kubernetes provides the rolling update feature to update your application with zero downtime.

When you perform a rolling update, TiDB Operator serially deletes an old Pod and creates the corresponding new Pod in the order of PD, TiKV, and TiDB. After the new Pod runs normally, TiDB Operator proceeds with the next Pod.

During the rolling update, TiDB Operator automatically completes Leader transfer for PD and TiKV. Under the highly available deployment topology (minimum requirements: PD * 3, TiKV * 3, TiDB * 2), performing a rolling update to PD and TiKV servers does not impact the running application. If your client supports retrying stale connections, performing a rolling update to TiDB servers does not impact application, either.

Upgrade steps

  1. In TidbCluster CR, modify the image configurations of all components of the cluster to be upgraded.

    kubectl edit tc ${cluster_name} -n ${namespace}
    

    Usually, all components in a cluster are in the same version. You can upgrade the TiDB cluster simply by modifying spec.version. If you need to use different versions for different components, modify spec.<pd/tidb/tikv/pump/tiflash/ticdc>.version.

    The version field has following formats:

    • spec.version: the format is imageTag, such as v6.1.0
    • spec.<pd/tidb/tikv/pump/tiflash/ticdc>.version: the format is imageTag, such as v3.1.0
  2. Check the upgrade progress:

    watch kubectl -n ${namespace} get pod -o wide
    

    After all the Pods finish rebuilding and become Running, the upgrade is completed.

Troubleshoot the upgrade

If the PD cluster is unavailable due to PD configuration errors, PD image tag errors, NodeAffinity, or other causes, you might not be able to successfully upgrade the TiDB cluster. In such cases, you can force an upgrade of the cluster to recover the cluster functionality.

The steps of force upgrade are as follows:

  1. Set annotation for the cluster:

    kubectl annotate --overwrite tc ${cluster_name} -n ${namespace} tidb.pingcap.com/force-upgrade=true
    
  2. Change the related PD configuration to make sure that PD turns into a normal state.

  3. After the PD cluster recovers, you must execute the following command to disable the forced upgrade; otherwise, an exception may occur in the next upgrade:

    kubectl annotate tc ${cluster_name} -n ${namespace} tidb.pingcap.com/force-upgrade-
    

After taking the steps above, your TiDB cluster recovers its functionality. You can upgrade the cluster normally.

Download PDFRequest docs changes
Was this page helpful?
Open Source Ecosystem
TiDB
TiKV
TiSpark
Chaos Mesh
© 2022 PingCAP. All Rights Reserved.