Advanced StatefulSet Controller

Kubernetes has a built-in StatefulSet that allocates consecutive serial numbers to Pods. For example, when there are three replicas, the Pods are named as pod-0, pod-1, and pod-2. When scaling out or scaling in, you must add a Pod at the end or delete the last pod. For example, when you scale out to four replicas, pod-3 is added. When you scale in to two replicas, pod-2 is deleted.

When you use local storage, Pods are associated with the Nodes storage resources and cannot be scheduled freely. If you want to delete one of the Pods in the middle to maintain its Node but no other Nodes can be migrated, or if you want to delete a Pod that fails and to create another Pod with a different serial number, you cannot implement such desired function by the built-in StatefulSet.

The advanced StatefulSet controller is implemented based on the built-in StatefulSet controller. It supports freely controlling the serial number of Pods. This document describes how to use the advanced StatefulSet controller in TiDB Operator.

Enable

1. Load the Advanced StatefulSet CRD file:

   • For Kubernetes versions < 1.16:

     kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/v1.3.10/manifests/advanced-statefulset-crd.v1beta1.yaml
     

   • For Kubernetes versions >= 1.16:

     kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/v1.3.10/manifests/advanced-statefulset-crd.v1.yaml
     

2. Enable the AdvancedStatefulSet feature in values.yaml of the TiDB Operator chart:

   features:
   - AdvancedStatefulSet=true
   advancedStatefulset:
     create: true
   

3. Upgrade TiDB Operator. For details, refer to Upgrade TiDB Operator.

4. After upgrading TiDB Operator, check the AdvancedStatefulSet Controller is deployed by the following command:

   kubectl get pods -n ${operator-ns} --selector app.kubernetes.io/component=advanced-statefulset-controller
   

   Expected output

   NAME                                               READY       STATUS    RESTARTS   AGE
   advanced-statefulset-controller-67885c5dd9-f522h   1/1         Running   0          10s
   

Usage

This section describes how to use the advanced StatefulSet controller.

View the AdvancedStatefulSet Object by kubectl

The data format of AdvancedStatefulSet is the same as that of StatefulSet, but AdvancedStatefulSet is implemented based on CRD, with asts as the alias. You can view the AdvancedStatefulSet object in the namespace by running the following command:

kubectl get -n ${namespace} asts

Specify the Pod to be scaled in

With the advanced StatefulSet controller, when scaling in TidbCluster, you can not only reduce the number of replicas, but also specify the scaling in of any Pod in the PD, TiDB, or TiKV components by configuring annotations.

For example:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: asts
spec:
  version: v6.1.0
  timezone: UTC
  pvReclaimPolicy: Delete
  pd:
    baseImage: pingcap/pd
    maxFailoverCount: 0
    replicas: 3
    requests:
      storage: "1Gi"
    config: {}
  tikv:
    baseImage: pingcap/tikv
    maxFailoverCount: 0
    replicas: 4
    requests:
      storage: "1Gi"
    config: {}
  tidb:
    baseImage: pingcap/tidb
    maxFailoverCount: 0
    replicas: 2
    service:
      type: ClusterIP
    config: {}

The above configuration deploys 4 TiKV instances, namely basic-tikv-0, basic-tikv-1, ..., basic-tikv-3. If you want to delete basic-tikv-1, set spec.tikv.replicas to 3 and configure the following annotations:

metadata:
  annotations:
    tikv.tidb.pingcap.com/delete-slots: '[1]'

The complete example is as follows:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  annotations:
    tikv.tidb.pingcap.com/delete-slots: '[1]'
  name: asts
spec:
  version: v6.1.0
  timezone: UTC
  pvReclaimPolicy: Delete
  pd:
    baseImage: pingcap/pd
    maxFailoverCount: 0
    replicas: 3
    requests:
      storage: "1Gi"
    config: {}
  tikv:
    baseImage: pingcap/tikv
    maxFailoverCount: 0
    replicas: 3
    requests:
      storage: "1Gi"
    config: {}
  tidb:
    baseImage: pingcap/tidb
    maxFailoverCount: 0
    replicas: 2
    service:
      type: ClusterIP
    config: {}

The supported annotations are as follows:

• pd.tidb.pingcap.com/delete-slots: Specifies the serial numbers of the Pods to be deleted in the PD component.
• tidb.tidb.pingcap.com/delete-slots: Specifies the serial number of the Pods to be deleted in the TiDB component.
• tikv.tidb.pingcap.com/delete-slots: Specifies the serial number of the Pods to be deleted in the TiKV component.

The value of Annotation is an integer array of JSON, such as [0], [0,1], [1,3].

Specify the location to scale out

You can reverse the above operation of scaling in to restore basic-tikv-1.

For example:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  annotations:
    tikv.tidb.pingcap.com/delete-slots: '[]'
  name: asts
spec:
  version: v6.1.0
  timezone: UTC
  pvReclaimPolicy: Delete
  pd:
    baseImage: pingcap/pd
    maxFailoverCount: 0
    replicas: 3
    requests:
      storage: "1Gi"
    config: {}
  tikv:
    baseImage: pingcap/tikv
    maxFailoverCount: 0
    replicas: 4
    requests:
      storage: "1Gi"
    config: {}
  tidb:
    baseImage: pingcap/tidb
    maxFailoverCount: 0
    replicas: 2
    service:
      type: ClusterIP
    config: {}

The delete-slots annotations can be left empty or deleted completely.