Detail
According to the documentation on Partition your flash devices, Aerospike recommends keeping the number of partitions usually at most equal to the number of CPU cores.
What could get impacted if we had more partitions than the number of CPU cores? Would it be a bad idea to have more partitions than the number of CPU cores?
Answer
This is a general guideline we have. To get the most optimal configuration, we highly recommend testing and benchmarking. There have been use cases where more than a device per core was beneficial. It could be different depending on the workload.
Having more partitions than CPU cores could cause Aerospike to run multiple swb buffers and write threads per core, which could be less optimal. Each core would have to switch contexts to write from two buffers per core.
NVMe disks typically have 4 dedicated PCIe lanes to the CPU, which means we can perform 4 operations at the same time to and from the disk without queuing on I/O (PCIe) bus.
Since Aerospike has a single swb per configured “device”, in order to allow Aerospike to perform up to 4 simultaneous writes to the disk at the same time, the disk should have 4 partitions. Otherwise, we may be bottlenecking on PCIe on writes as we would only have one “in” lane. Reads should not be impacted.
In the case of hyperthreading, Aerospike would consider hyperthread cores as cores and would create as many swb buffers as partitions. The hardware and kernel could behave poorly, because each physical core can handle only one I/O at a time even when hyperthreading. One virtual core is blocked by another doing I/O when they are on the same physical core. For example, 32 cores would still mean max 32 I/O operations at the same time regardless of whether hyperthreading is enabled or not.
That is why Aerospike usually recommends keeping the number of partitions equal to the number of CPU cores.