Sign InTry Free

Prerequisites for TiDB on Kubernetes

This document introduces the hardware and software prerequisites for deploying a TiDB cluster on Kubernetes.

Software version

Software NameVersion
DockerDocker CE 18.09.6
CentOS7.6 and kernel 3.10.0-957 or later

Configure the firewall

It is recommended that you disable the firewall.

systemctl stop firewalld systemctl disable firewalld

If you cannot stop the firewalld service, to ensure the normal operation of Kubernetes, take the following steps:

  1. Enable the following ports on the master, and then restart the service:

    firewall-cmd --permanent --add-port=6443/tcp firewall-cmd --permanent --add-port=2379-2380/tcp firewall-cmd --permanent --add-port=10250/tcp firewall-cmd --permanent --add-port=10251/tcp firewall-cmd --permanent --add-port=10252/tcp firewall-cmd --permanent --add-port=10255/tcp firewall-cmd --permanent --add-port=8472/udp firewall-cmd --add-masquerade --permanent # Set it when you need to expose NodePort on the master node. firewall-cmd --permanent --add-port=30000-32767/tcp systemctl restart firewalld
  2. Enable the following ports on the nodes, and then restart the service:

    firewall-cmd --permanent --add-port=10250/tcp firewall-cmd --permanent --add-port=10255/tcp firewall-cmd --permanent --add-port=8472/udp firewall-cmd --permanent --add-port=30000-32767/tcp firewall-cmd --add-masquerade --permanent systemctl restart firewalld

Configure Iptables

The FORWARD chain is configured to ACCEPT by default and is set in the startup script:


Disable SELinux

setenforce 0 sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Disable swap

To make kubelet work, you need to turn off swap and comment out the swap-related line in the /etc/fstab file.

swapoff -a sed -i 's/^\(.*swap.*\)$/#\1/' /etc/fstab

Configure kernel parameters

Configure the kernel parameters as follows. You can also adjust them according to your environment:

modprobe br_netfilter cat <<EOF > /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-arptables = 1 net.core.somaxconn = 32768 vm.swappiness = 0 net.ipv4.tcp_syncookies = 0 net.ipv4.ip_forward = 1 fs.file-max = 1000000 fs.inotify.max_user_watches = 1048576 fs.inotify.max_user_instances = 1024 net.ipv4.conf.all.rp_filter = 1 net.ipv4.neigh.default.gc_thresh1 = 80000 net.ipv4.neigh.default.gc_thresh2 = 90000 net.ipv4.neigh.default.gc_thresh3 = 100000 EOF sysctl --system

Configure the Irqbalance service

The Irqbalance service binds the interrupts of each equipment to different CPUs respectively. This avoids the performance bottleneck when all interrupt requests are sent to the same CPU.

systemctl enable irqbalance systemctl start irqbalance

Configure the CPUfreq governor mode

To make full use of CPU performance, set the CPUfreq governor mode to performance. For details, see Configure the CPUfreq governor mode on the target machine.

cpupower frequency-set --governor performance

Configure ulimit

The TiDB cluster uses many file descriptors by default. The ulimit of the worker node must be greater than or equal to 1048576.

cat <<EOF >> /etc/security/limits.conf root soft nofile 1048576 root hard nofile 1048576 root soft stack 10240 EOF sysctl --system

Docker service

It is recommended to install Docker CE 18.09.6 or later versions. See Install Docker for details.

After the installation, take the following steps:

  1. Save the Docker data to a separate disk. The data mainly contains images and the container logs. To implement this, set the --data-root parameter:

    cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ], "data-root": "/data1/docker" } EOF

    The above command sets the data directory of Docker to /data1/docker.

  2. Set ulimit for the Docker daemon:

    1. Create the systemd drop-in directory for the docker service:

      mkdir -p /etc/systemd/system/docker.service.d
    2. Create a file named as /etc/systemd/system/docker.service.d/limit-nofile.conf, and configure the value of the LimitNOFILE parameter. The value must be a number equal to or greater than 1048576.

      cat > /etc/systemd/system/docker.service.d/limit-nofile.conf <<EOF [Service] LimitNOFILE=1048576 EOF
    3. Reload the configuration.

      systemctl daemon-reload && systemctl restart docker

Kubernetes service

To deploy a multi-master, highly available cluster, see Kubernetes documentation.

The configuration of the Kubernetes master depends on the number of nodes. More nodes consumes more resources. You can adjust the number of nodes as needed.

Nodes in a Kubernetes clusterKubernetes master configuration
1-51vCPUs 4GB Memory
6-102vCPUs 8GB Memory
11-1004vCPUs 16GB Memory
101-2508vCPUs 32GB Memory
251-50016vCPUs 64GB Memory
501-500032vCPUs 128GB Memory

After Kubelet is installed, take the following steps:

  1. Save the Kubelet data to a separate disk (it can share the same disk with Docker). The data mainly contains the data used by emptyDir. To implement this, set the --root-dir parameter:

    echo "KUBELET_EXTRA_ARGS=--root-dir=/data1/kubelet" > /etc/sysconfig/kubelet systemctl restart kubelet

    The above command sets the data directory of Kubelet to /data1/kubelet.

  2. Reserve compute resources by using Kubelet, to ensure that the system process of the machine and the kernel process of Kubernetes have enough resources for operation in heavy workloads. This maintains the stability of the entire system.

TiDB cluster's requirements for resources

To determine the machine configuration, see Server recommendations.

In a production environment, avoid deploying TiDB instances on a kubernetes master, or deploy as few TiDB instances as possible. Due to the NIC bandwidth, if the NIC of the master node works at full capacity, the heartbeat report between the worker node and the master node will be affected and might lead to serious problems.

Was this page helpful?

Download PDFRequest docs changesAsk questions on Discord
One-stop & interactive experience of TiDB's capabilities WITHOUT registration.
TiDB Dedicated
TiDB Serverless
Get Demo
Get Started
© 2024 PingCAP. All Rights Reserved.
Privacy Policy.