Problem Description
An Aerospike cluster with Strong Consistency namespaces goes through two or more cluster changes in quick succession, most often from network instability affecting several nodes. Partitions go unavailable, and once the roster is complete again they are reported as dead.
Explanation
Why SC tracks a regime
During a cluster change there is a short period where more than one node can believe it is master for the same partition. Timestamps are not enough to tell whose writes are current, so the cluster tracks a regime number. The outgoing master keeps the older value and the incoming master's writes carry the newer one. A write accepted by a node that had not yet processed the cluster change is discarded when it meets a higher regime.
The regime advances every time the cluster reforms. That is normal and visible in the log.
What a double pump is
Reforming the cluster starts with an exchange. Every node sends its partition version data to the others, and the principal commits once all of them have acknowledged. A node that has heard from all the others advances its regime at that point.
If a second cluster change lands after that but before every node has accepted the principal's commit, the exchange is abandoned and restarted, and the regime advances again. That is a double pump. The window is the whole exchanging phase, and it closes only when every node has passed through ready-to-commit and returned to rest.
The affected partitions are marked as subsets rather than full copies.
Why subsets cost availability
A subset copy is normally not counted when SC works out whether a partition is available. If both the roster master and the roster replica for a partition are subsets, the partition goes unavailable.
The exception is a super majority. If fewer nodes are missing than the replication factor, a combination of subsets is enough and the partition stays active.
Two kinds of node are excluded from that count even when they are back in the cluster: a node with empty drives or no data, and a node that did not shut down cleanly, unless commit-to-device is enabled. Those carry an evade flag until they are properly inducted with all of their data. This is why a cluster can look complete and still be short of a super majority.
A partition that is unavailable while every node in the roster is present is, by definition, dead. Dead is not a claim that data was lost. It means the cluster will not decide on its own whether the partition is consistent, and is waiting for an operator to decide.
Confirming it
In the example below a single reformation advances the regime by 2:
Nov 01 2023 15:39:22 GMT: INFO (partition): (partition_balance_ee.c:1000) {myns} 22 of 22 nodes participating - regime 475 -> 477A double pump advances it by 4:
Nov 01 2023 15:41:24 GMT: INFO (partition): (partition_balance_ee.c:1000) {myns} 22 of 22 nodes participating - regime 477 -> 481Increments of 6 or more occur. An aborting partition exchange line appears in the same period. Whether the jump is 4 or 6 makes no difference to the handling.
Racks
A double pump is not limited to rack-aware clusters. Rack layout matters afterwards, through the roster.
Rack placement decides which nodes hold the roster replicas for a partition, and losing a whole rack removes all of those nodes from the roster at once. On two racks that is exactly half the roster, which is not a majority, so partitions whose roster master sat on the lost rack go unavailable.
Solution
Dead partitions are brought back with revive followed by recluster. Revive on its own does nothing. The recluster is what applies it.
Admin+> manage revive ns test
Admin+> manage reclusterThe decision is the operator's. Inspect what led to the dead partitions and consider whether writes were actually lost before reviving. If nodes were lost uncleanly at the same time, or came back with empty storage, treat lost writes as possible. In most instances of a double pump it is safe to revive once whatever caused the rapid cluster changes has been dealt with.
From version 7.1, auto-revive will revive dead partitions at startup without waiting for an operator. It trades the dead-partition signal for availability. If writes were lost during an unclean shutdown, auto-revive accepts that silently, and it will not revive at all if a storage device was erased.
Notes
While the cluster is still missing nodes the affected partitions are subsets and are reported as unavailable, not dead. They only become dead once the full roster is present again, either because the missing nodes returned or because the roster was reset.
Applies To Earliest Version
4.0
Applies To Latest Version
Current Version