Context
Overview:
Automatic NUMA Balancing is a kernel feature that periodically moves tasks closer to the memory they are accessing and migrates memory pages to nodes local to the tasks. While beneficial for general workloads, it can cause latency spikes in high-performance computing (HPC), large databases, or real-time applications.
Method
1. Quick Check
Before making changes, verify the current status of NUMA balancing.
- Command:
sysctl kernel.numa_balancing - Values:
1(Enabled),0(Disabled)
2. Implementation Methods
Method A: Runtime (Temporary)
Use this for testing or immediate effect. Settings will be lost after a reboot.
# Using sysctl
sudo sysctl -w kernel.numa_balancing=0
# OR using the proc filesystem
echo 0 | sudo tee /proc/sys/kernel/numa_balancing
Method B: Persistent (Recommended)
Use this to ensure the setting survives a system reboot. This is the standard approach for production servers.
- Create a config file:
sudo vi /etc/sysctl.d/99-disable-numa.conf - Add the following line:
kernel.numa_balancing = 0 - Apply changes:
sudo sysctl -p /etc/sysctl.d/99-disable-numa.conf
Method C: Kernel Boot Parameter
This disables the feature at the earliest possible stage of the boot process.
- Edit GRUB: Update
/etc/default/gruband appendnuma_balancing=disableto theGRUB_CMDLINE_LINUXvariable. - Rebuild GRUB config:
# For BIOS/UEFI on RHEL 9 sudo grubby --update-kernel=ALL --args="numa_balancing=disable"
3. Verification & Troubleshooting
To confirm the change has been applied successfully, use the following tools:
| Tool | Command | Expected Result |
|---|---|---|
| sysctl | sysctl kernel.numa_balancing | 0 |
| numastat | numastat -n | Monitor for reduced "interleave" hits |
| dmesg | `dmesg | grep -i numa` |
4. When to Disable
- Low Latency Requirements: When micro-stutters from page migration are unacceptable.
- Static Pinning: If you are already manually pinning CPUs and memory using
numactlorcpusets. - Specific Database Recommendations: When vendor documentation explicitly requires it.