Problem Description
Stop writes were triggered by breaching stop-writes-used-pct (7.x), max-used-pct (6.x and below) even though our cluster is using a 10TiB device. stop-writes-used-pct is set to 70, but only around 1.4TiB is being used.
Example namespace config:
namespace test {
replication-factor 2
memory-size 64G
storage-engine device {
device /dev/mydevice // mydevice is 10TiB
write-block-size 128K
}
}
Explanation
While Aerospike does support devices larger than 2TiB, we do require you to partition the physical device into smaller partitions. The max size per partition is 2TiB. In the above example, we can see that stop writes would get triggered since we would only be using 2TiB of the 10TiB device. With stop-writes-used-pct set to 70 (default), 1.4TiB used would be 70% of 2TiB. That is why stop writes were triggered in this case.
Solution
Partition the larger device into smaller partitions. Each partition can be at maximum 2TiB. You can refer to the following documentations for more details and examples: