Hybrid Deployment Topology
This document describes the topology and key parameters of the TiKV and TiDB hybrid deployment.
The hybrid deployment is usually used in the following scenario:
The deployment machine has multiple CPU processors with sufficient memory. To improve the utilization rate of the physical machine resources, multiple instances can be deployed on a single machine, that is, TiDB and TiKV's CPU resources are isolated through NUMA node bindings. PD and Prometheus are deployed together, but their data directories need to use separate file systems.
|Instance||Count||Physical machine configuration||IP||Configuration|
|TiDB||6||32 VCore 64GB||10.0.1.1|
|Configure NUMA to bind CPU cores|
|PD||3||16 VCore 32 GB||10.0.1.4|
|Configure the |
|TiKV||6||32 VCore 64GB||10.0.1.7|
|1. Separate the instance-level port and status_port; |
2. Configure the global parameters
3. Configure labels of the instance-level host;
4. Configure NUMA to bind CPU cores
|Monitoring & Grafana||1||4 VCore 8GB * 1 500GB (ssd)||10.0.1.10||Default configuration|
For detailed descriptions of the configuration items in the above TiDB cluster topology file, see Topology Configuration File for Deploying TiDB Using TiUP.
This section introduces the key parameters when you deploy multiple instances on a single machine, which is mainly used in scenarios when multiple instances of TiDB and TiKV are deployed on a single machine. You need to fill in the results into the configuration template according to the calculation methods provided below.
Optimize the configuration of TiKV
readpoolto be self-adaptive to the thread pool. By configuring the
readpool.unified.max-thread-countparameter, you can make
readpool.coprocessorshare a unified thread pool, and set the self-adaptive switch respectively.
readpool.storage.use-unified-pool: true readpool.coprocessor.use-unified-pool: true
The calculation method:
readpool.unified.max-thread-count = cores * 0.8 / the number of TiKV instances
To configure the storage CF (all RocksDB column families) to be self-adaptive to memory. By configuring the
storage.block-cache.capacityparameter, you can make CF automatically balance the memory usage.
storage.block-cacheenables the CF self-adaptation by default. You do not need to modify it.
The calculation method:
storage.block-cache.capacity = (MEM_TOTAL * 0.5 / the number of TiKV instances)
If multiple TiKV instances are deployed on the same physical disk, add the
capacityparameter in the TiKV configuration:
raftstore.capacity = disk total capacity / the number of TiKV instances
The label scheduling configuration
Since multiple instances of TiKV are deployed on a single machine, if the physical machines go down, the Raft Group might lose two of the default three replicas, which causes the cluster unavailability. To address this issue, you can use the label to enable the smart scheduling of PD, which ensures that the Raft Group has more than two replicas in multiple TiKV instances on the same machine.
The TiKV configuration
The same host-level label information is configured for the same physical machine:
config: server.labels: host: tikv1
The PD configuration
To enable PD to identify and scheduling Regions, configure the labels type for PD:
pd: replication.location-labels: ["host"]
In the instance parameter module, configure the corresponding
numa_nodeparameter and add the number of CPU cores.
Before using NUMA to bind cores, make sure that the numactl tool is installed, and confirm the information of CPUs in the physical machines. After that, configure the parameters.
numa_nodeparameter corresponds to the