Hybrid Deployment Topology

This document describes the topology and key parameters of the TiKV and TiDB hybrid deployment.

The hybrid deployment is usually used in the following scenario:

The deployment machine has multiple CPU processors with sufficient memory. To improve the utilization rate of the physical machine resources, multiple instances can be deployed on a single machine, that is, TiDB and TiKV's CPU resources are isolated through NUMA node bindings. PD and Prometheus are deployed together, but their data directories need to use separate file systems.

Topology information

Topology templates

• The simple template for the hybrid deployment
• The complex template for the hybrid deployment

For detailed descriptions of the configuration items in the above TiDB cluster topology file, see Topology Configuration File for Deploying TiDB Using TiUP.

Key parameters

This section introduces the key parameters when you deploy multiple instances on a single machine, which is mainly used in scenarios when multiple instances of TiDB and TiKV are deployed on a single machine. You need to fill in the results into the configuration template according to the calculation methods provided below.

• Optimize the configuration of TiKV

  • To configure readpool to be self-adaptive to the thread pool. By configuring the readpool.unified.max-thread-count parameter, you can make readpool.storage and readpool.coprocessor share a unified thread pool, and set the self-adaptive switch respectively.

    • Enable readpool.storage and readpool.coprocessor:

      readpool.storage.use-unified-pool: true
      readpool.coprocessor.use-unified-pool: true
      

    • The calculation method:

      readpool.unified.max-thread-count = cores * 0.8 / the number of TiKV instances
      

  • To configure the storage CF (all RocksDB column families) to be self-adaptive to memory. By configuring the storage.block-cache.capacity parameter, you can make CF automatically balance the memory usage.

    • The calculation method:

      storage.block-cache.capacity = (MEM_TOTAL * 0.5 / the number of TiKV instances)
      

  • If multiple TiKV instances are deployed on the same physical disk, add the capacity parameter in the TiKV configuration:

    raftstore.capacity = disk total capacity / the number of TiKV instances
    

• The label scheduling configuration

  When multiple TiKV instances are deployed on a single physical machine, a machine failure might cause a Raft Group to lose two of its default three replicas. Losing these replicas makes that Raft Group unavailable. Configure host-level labels so that PD can spread replicas across different physical machines and reduce this risk.

  • The TiKV configuration

    The same host-level label information is configured for the same physical machine:

    config:
      server.labels:
        host: tikv1
    

  • The PD configuration

    To enable PD to identify and scheduling Regions, configure the labels type for PD:

    pd:
      replication.location-labels: ["host"]
    

• numa_node core binding

  • In the instance configuration, set numa_node to the NUMA node ID or IDs, such as "0" or "0,1".

  • Before using NUMA to bind cores, make sure that the numactl tool is installed, and confirm the information of CPUs in the physical machines. After that, configure the parameters.

  • After you configure numa_node, TiUP uses numactl to bind the CPU and memory resources of the instance to the specified NUMA nodes.