Customize the Configuration of Kubernetes Native Resources

This document describes how to use the Overlay feature of TiDB Operator to customize Kubernetes native resource configurations.

The Overlay feature is a configuration mechanism in TiDB Operator that enables you to customize the configuration of native resources in Kubernetes clusters (such as Pods and PersistentVolumeClaims (PVCs)) without modifying the TiDB Operator source code. This enables you to meet specific deployment requirements.

Supported resource types

Currently, TiDB Operator supports customizing the following resource types through the Overlay feature:

• Pod: modify the Pod metadata (such as labels and annotations) and the spec field.
• PersistentVolumeClaim (PVC): modify PVC metadata (such as labels and annotations).

Usage

Customize Pod configurations (Pod Overlay)

Pod Overlay enables you to modify the Pod metadata (such as labels and annotations) and the spec field. You can configure this using the spec.template.spec.overlay.pod field in the Custom Resource (CR) of a component group (such as PDGroup, TiDBGroup, TiKVGroup, TiFlashGroup, TiProxyGroup, or TiCDCGroup).

The following example shows how to add an environment variable named CUSTOM_ENV_VAR to the PD container:

apiVersion: core.pingcap.com/v1alpha1
kind: PDGroup
metadata:
  name: pd
spec:
  template:
    spec:
      overlay:
        pod:
          spec:
            containers:
              - name: pd
                env:
                  - name: "CUSTOM_ENV_VAR"
                    value: "custom_value"

Customize PVC configurations (PVC Overlay)

PVC Overlay enables you to modify the PVC metadata (such as labels and annotations) and the spec field. You can configure this using the spec.template.spec.overlay.volumeClaims field in the Custom Resource (CR) of a component group (such as PDGroup, TiDBGroup, TiKVGroup, TiFlashGroup, TiProxyGroup, or TiCDCGroup).

The following example shows how to add a custom label named custom-label to the PVC of the TiKV component:

apiVersion: core.pingcap.com/v1alpha1
kind: TiKVGroup
metadata:
  name: tikv
spec:
  template:
    spec:
      volumes:
      - name: data
        mounts:
          - type: data
        storage: 100Gi
      overlay:
        volumeClaims:
          - name: data
            volumeClaim:
              metadata:
                labels:
                  custom-label: "value"

Notes

When using the Overlay feature, note the following:

• The Overlay feature merges the configuration you define in the spec.template.spec.overlay field with the default configuration automatically generated by TiDB Operator. If conflicts exist, the configuration defined in Overlay takes precedence.
• For map-type fields such as nodeSelector and labels, the Overlay feature appends the key-value pairs you define while preserving existing configurations, rather than replacing them entirely.
• Before modifying configurations using the Overlay feature, ensure that you fully understand the potential impact of these changes, especially for critical configurations such as securityContext and resource limits.
• The fields that can be modified through Overlay depend on the Kubernetes API version used by TiDB Operator.

Example use cases

This section provides common examples of using the Overlay feature to help you understand and apply it for customizing Kubernetes resource configurations.

Configure Pod resource limits and affinity

The following example shows how to use Overlay to configure resource request limits and affinity for TiDB Pods:

spec:
  template:
    spec:
      overlay:
        pod:
          spec:
            containers:
              - name: tidb
                resources:
                  limits:
                    cpu: "4"
                    memory: "8Gi"
                  requests:
                    cpu: "2"
                    memory: "4Gi"
            affinity:
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                  - matchExpressions:
                    - key: dedicated
                      operator: In
                      values:
                      - tidb

Configure Pod security context

The following example shows how to configure Pod sysctls parameters using Overlay:

spec:
  template:
    spec:
      overlay:
        pod:
          spec:
            securityContext:
              sysctls:
              - name: net.core.somaxconn
                value: "1024"

Update Pod and PVC labels or annotations in place

Using Overlay, you can update Pod and PVC labels or annotations without restarting the Pod:

spec:
  template:
    spec:
      overlay:
        pod:
          metadata:
            labels:
              custom-label: "value"
            annotations:
              custom-annotation: "value"
        volumeClaims:
          - name: data
            volumeClaim:
              metadata:
                labels:
                  custom-label: "value"
                annotations:
                  custom-annotation: "value"

Inject a sidecar container

You can use Overlay to inject a sidecar container into a Pod for purposes such as monitoring or log collection:

spec:
  template:
    spec:
      overlay:
        pod:
          spec:
            initContainers:
              - name: logshipper
                image: alpine:latest
                restartPolicy: Always
                command: ['sh', '-c', 'tail -F /opt/logs.txt']