Detail
A cluster has a non-zero value set for migrate-fill-delay however in the event of a node removal the aerospike.log shows partitions to be migrated as follows:
Jun 04 2019 13:56:01 GMT: INFO (partition): (partition_balance.c:957) rebalanced: expected-migrations (1340,1389,1340) fresh-partitions 0
Why are these messages shown?
Answer
The answer lies in the two part mechanism by which an Aerospike cluster will rebalance after a cluster event.
- The first part of that is the cluster determines which partitions should go to which nodes in the new cluster state.
- The second part of the process is the movement of partitions between nodes to reflect the partition mapping decision.
When migrate-fill-delay is set to a non-zero value it inserts the specified delay between these two steps. The cluster decides where the partitions will go and then waits for migrate-fill-delay until it actually moves the data.
This will be reflected in a subsequent log line as follows:
Jun 04 2019 13:56:04 GMT: INFO (info): (ticker.c:454) {test} migrations: remaining (1340,1389,1340) active (0,0,0) complete-pct 0.00
The line is stating that there are pending migrations but that there are none active as the migrations have been delayed.
The product is working as designed.