Problem Description
When I check for the replication state of records in a strong consistency namespace, I noticed that there are some records in unreplicated state.asadm > show stat like unreplicated_records
Explanation
I follow the UDF Peek method in KB How to trigger re-replication for unreplicated records within a strongly consistent Aerospike namespace? in order to clear out the unreplicated state but it does a no-op without any re-replication and the count of un-replicated recordsdoes not drop.Solution
One situation that could trigger this situation is if the un-replicated records are replicas and not masters.With replication factor of 3, there are ways prole can be un-replicated due to the extra signaling step for the master.
With replication factor 2, creating un-replicated replicas may happen however in a much complex manner.
The condition would be that those replicas might have been acting master at some point and got in un-replicated state then. During migrations we wouldn’t set the replication state as we are assuming we go from master to prole so those acting masters would have replicated even though they themselves in unreplicated state.
We would then end up with the eventual master as clean, but the eventual prole as unreplicated. This would trigger the unreplicated statistic showing those and since UDF Peek looks at the masters these would be not triggering re-replication.
Following the Scan-Touch approach for such cases will fix the state of the unreplicated proles as all records would be touched and replicated