Prerequisites for TiDB on Kubernetes

This document introduces the hardware and software prerequisites for deploying a TiDB cluster on Kubernetes.

Software version

Configure the firewall

It is recommended that you disable the firewall.

systemctl stop firewalld
systemctl disable firewalld

If you cannot stop the firewalld service, to ensure the normal operation of Kubernetes, take the following steps:

1. Enable the following ports on the master, and then restart the service:

   firewall-cmd --permanent --add-port=6443/tcp
   firewall-cmd --permanent --add-port=2379-2380/tcp
   firewall-cmd --permanent --add-port=10250/tcp
   firewall-cmd --permanent --add-port=10251/tcp
   firewall-cmd --permanent --add-port=10252/tcp
   firewall-cmd --permanent --add-port=10255/tcp
   firewall-cmd --permanent --add-port=8472/udp
   firewall-cmd --add-masquerade --permanent
   
   # Set it when you need to expose NodePort on the master node.
   firewall-cmd --permanent --add-port=30000-32767/tcp
   systemctl restart firewalld
   

2. Enable the following ports on the nodes, and then restart the service:

   firewall-cmd --permanent --add-port=10250/tcp
   firewall-cmd --permanent --add-port=10255/tcp
   firewall-cmd --permanent --add-port=8472/udp
   firewall-cmd --permanent --add-port=30000-32767/tcp
   firewall-cmd --add-masquerade --permanent
   
   systemctl restart firewalld
   

Configure Iptables

The FORWARD chain is configured to ACCEPT by default and is set in the startup script:

iptables -P FORWARD ACCEPT

Disable SELinux

setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Disable swap

To make kubelet work, you need to turn off swap and comment out the swap-related line in the /etc/fstab file.

swapoff -a
sed -i 's/^\(.*swap.*\)$/#\1/' /etc/fstab

Configure kernel parameters

Configure the kernel parameters as follows. You can also adjust them according to your environment:

modprobe br_netfilter

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
net.core.somaxconn = 32768
vm.swappiness = 0
net.ipv4.tcp_syncookies = 0
net.ipv4.ip_forward = 1
fs.file-max = 1000000
fs.inotify.max_user_watches = 1048576
fs.inotify.max_user_instances = 1024
net.ipv4.conf.all.rp_filter = 1
net.ipv4.neigh.default.gc_thresh1 = 80000
net.ipv4.neigh.default.gc_thresh2 = 90000
net.ipv4.neigh.default.gc_thresh3 = 100000
EOF

sysctl --system

Configure the Irqbalance service

The Irqbalance service binds the interrupts of each equipment to different CPUs respectively. This avoids the performance bottleneck when all interrupt requests are sent to the same CPU.

systemctl enable irqbalance
systemctl start irqbalance

Configure the CPUfreq governor mode

To make full use of CPU performance, set the CPUfreq governor mode to performance. For details, see Configure the CPUfreq governor mode on the target machine.

cpupower frequency-set --governor performance

Configure ulimit

The TiDB cluster uses many file descriptors by default. The ulimit of the worker node must be greater than or equal to 1048576.

cat <<EOF >>  /etc/security/limits.conf
root        soft        nofile        1048576
root        hard        nofile        1048576
root        soft        stack         10240
EOF
sysctl --system

Docker service

It is recommended to install Docker CE 18.09.6 or later versions. See Install Docker for details.

After the installation, take the following steps:

1. Save the Docker data to a separate disk. The data mainly contains images and the container logs. To implement this, set the --data-root parameter:

   cat > /etc/docker/daemon.json <<EOF
   {
     "exec-opts": ["native.cgroupdriver=systemd"],
     "log-driver": "json-file",
     "log-opts": {
       "max-size": "100m"
     },
     "storage-driver": "overlay2",
     "storage-opts": [
       "overlay2.override_kernel_check=true"
     ],
     "data-root": "/data1/docker"
   }
   EOF
   

   The above command sets the data directory of Docker to /data1/docker.

2. Set ulimit for the Docker daemon:

   1. Create the systemd drop-in directory for the docker service:

      mkdir -p /etc/systemd/system/docker.service.d
      

   2. Create a file named as /etc/systemd/system/docker.service.d/limit-nofile.conf, and configure the value of the LimitNOFILE parameter. The value must be a number equal to or greater than 1048576.

      cat > /etc/systemd/system/docker.service.d/limit-nofile.conf <<EOF
      [Service]
      LimitNOFILE=1048576
      EOF
      

      Note

      DO NOT set the value of LimitNOFILE to infinity. Due to a bug of systemd, the infinity value of systemd in some versions is 65536.

   3. Reload the configuration.

      systemctl daemon-reload && systemctl restart docker
      

Kubernetes service

To deploy a multi-master, highly available cluster, see Kubernetes documentation.

The configuration of the Kubernetes master depends on the number of nodes. More nodes consumes more resources. You can adjust the number of nodes as needed.

After Kubelet is installed, take the following steps:

1. Save the Kubelet data to a separate disk (it can share the same disk with Docker). The data mainly contains the data used by emptyDir. To implement this, set the --root-dir parameter:

   echo "KUBELET_EXTRA_ARGS=--root-dir=/data1/kubelet" > /etc/sysconfig/kubelet
   systemctl restart kubelet
   

   The above command sets the data directory of Kubelet to /data1/kubelet.

2. Reserve compute resources by using Kubelet, to ensure that the system process of the machine and the kernel process of Kubernetes have enough resources for operation in heavy workloads. This maintains the stability of the entire system.

TiDB cluster's requirements for resources

To determine the machine configuration, see Server recommendations.

In a production environment, avoid deploying TiDB instances on a kubernetes master, or deploy as few TiDB instances as possible. Due to the NIC bandwidth, if the NIC of the master node works at full capacity, the heartbeat report between the worker node and the master node will be affected and might lead to serious problems.