Articles in this section

How to set CPU requests for Aerospike containers?

Audience: Kubernetes administrators running Aerospike with the Aerospike Kubernetes Operator (AKO).

Applies to: Aerospike Kubernetes Operator 3.0.0 and later.


Context

Kubernetes puts every pod in a QoS class, based on the CPU and memory requests and limits of its containers:

  • BestEffort: no container in the pod has a request or a limit.
  • Burstable: at least one container has a request or a limit, but the pod does not meet the Guaranteed rules.
  • Guaranteed: every container has CPU and memory requests equal to its limits.

The QoS class decides how much CPU a pod gets when the node's CPUs are busy. The kernel shares contended CPU time by cgroup weight. A BestEffort pod gets the smallest weight there is: cpu.weight 1 on cgroup v2 (cpu.shares 2 on cgroup v1). A pod with a CPU request gets a weight that grows with its request.

AKO does not add requests or limits by default, so an AerospikeCluster created without resources runs as BestEffort, unless a sidecar sets its own resources or a LimitRange in the namespace adds default requests. Most of the time that does not show, because the node has idle CPU. When another process on the node bursts, for example a monitoring agent, a security agent or a burst of short-lived scripts, it takes the CPU first. Aerospike's service threads wait until it is done, and client requests can time out. The usual sign is client timeouts that land on the same second of every minute, while Aerospike's own CPU use stays flat.

Giving the Aerospike container a CPU request makes the pod Burstable and raises its share against the other pods on the node. It does not change its share against processes that run outside Kubernetes, such as host services and their scripts: those compete with all pods together, at a level a pod's request does not affect. Before relying on a request, check where the bursting process runs, for example with cat /proc/<pid>/cgroup on the node: a path under kubepods is a pod, a path under system.slice is a host service.


Method

1. Check the current state

The pod's QoS class:

kubectl -n <namespace> get pod <aerospike-pod> -o jsonpath='{.status.qosClass}{"\n"}'

What the AerospikeCluster sets today (empty output means nothing):

kubectl -n <namespace> get aerospikecluster <cluster> -o jsonpath='{.spec.podSpec.aerospikeContainer.resources}{"\n"}'

How much CPU each node can still hand out. The request must fit in the node's allocatable CPU after the requests of the other pods on it:

kubectl describe node <node> | sed -n '/Allocatable/,/Events/p'

2. Choose the value

  • Request about the CPU that Aerospike uses at its busiest, plus some headroom. The process: cpu-pct line in the Aerospike log shows Aerospike's CPU every 10 seconds, in percent of one CPU (450 means 4.5 CPUs).
  • Set requests only, and leave limits unset. The request alone sets the share. A CPU limit adds a CFS quota, which throttles Aerospike when it uses more than the limit in a scheduling period and can cause short stalls of its own. A container that goes over a memory limit is killed.
  • If you do set limits, AKO's webhook needs both a CPU limit and a memory limit, each at least as large as its request. A CPU request with only a memory limit is rejected, because the missing CPU limit counts as zero.

3. Apply it

Set the request in the AerospikeCluster resource, for example 2 CPUs:

spec:
  podSpec:
    aerospikeContainer:
      resources:
        requests:
          cpu: "2"

Or patch it:

kubectl -n <namespace> patch aerospikecluster <cluster> --type merge -p '{"spec":{"podSpec":{"aerospikeContainer":{"resources":{"requests":{"cpu":"2"}}}}}}'

This changes the pod specification. AKO replaces the pods one at a time by default (more at once if rollingUpdateBatchSize is set): each pod is deleted and created again, and Aerospike on the new pod does a cold start. AKO 4.6.0 and later can avoid the cold start on Aerospike 8.2 and later, when index-checkpoint-path is set, by saving an index checkpoint before deleting the pod. Plan for the cold-start time of your largest node, make the change in a quiet period, and try it on a test cluster first. Follow the rollout with:

kubectl -n <namespace> get pods -w

4. Verify

The pod should now be Burstable:

kubectl -n <namespace> get pod <aerospike-pod> -o jsonpath='{.status.qosClass}{"\n"}'

To read the weight itself on cgroup v2, open a debug pod on the node and read the pod's cgroup:

kubectl debug node/<node> -it --profile=sysadmin --image=busybox:1.36 -- sh

If your kubectl does not support --profile=sysadmin, leave the option out. The debug pod stays after you exit: delete the node-debugger-... pod when you are done.

The path depends on the kubelet's cgroup driver. With the systemd driver, which most managed Kubernetes services use, the pod's UID appears with dashes replaced by underscores:

cat /host/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<uid_with_underscores>.slice/cpu.weight

With the cgroupfs driver, the UID keeps its dashes:

cat /host/sys/fs/cgroup/kubepods/burstable/pod<uid>/cpu.weight

Get the pod's UID with kubectl -n <namespace> get pod <aerospike-pod> -o jsonpath='{.metadata.uid}'.

Then check that the timeouts that prompted the change have dropped.


Going further: Guaranteed pods with dedicated CPUs

A CPU request gives Aerospike a fair share. To keep other pods off Aerospike's CPUs altogether:

  • Make the pod Guaranteed: CPU and memory requests equal to limits on every container in the pod, with a whole number of CPUs for the Aerospike container. Kubernetes counts init containers too, so this includes AKO's init container (spec.podSpec.aerospikeInitContainer.resources), any containers in spec.podSpec.initContainers, and any sidecars, such as the Prometheus exporter.
  • Run the kubelet with the static CPU manager policy (cpuManagerPolicy: static). Guaranteed containers with whole CPUs then get those CPUs to themselves.

On managed Kubernetes the CPU manager policy is a node pool setting, and changing it usually means creating a new node pool. Processes outside Kubernetes, such as system services, can still run on those CPUs. To keep them off, pin them to other CPUs at the operating-system level, for example with systemd's CPUAffinity.

Was this article helpful?
0 out of 0 found this helpful