Problem Description
When I enable XDR with a rolling restart, I see all records on the enabled namespace made it to the destination. This was not the behavior in pre 5.0 and only newly created records would make it to the destination.
Explanation
XDR 5.0 has a different behaviour when adding a new dc dynamically and when adding via rolling restart.
Behaviour when adding a new dc via rolling restart.
XDR maintains a last ship time for each DC within the system metadata directory (SMD) of each node. This is necessary as from Aerospike 5.0 there is no file based logging and incoming digests are held in in-memory queues prior to shipment. At start up, the node checks the last ship time for each configured DC and ships anything later than that last ship time. When a new destination is added while the node is down, XDR just ships from the beginning. The reason that recoveries metric increases is that the shipping described here is done in recovery mode whereby XDR reduces a partition rather than shipping from entries in an in-memory queue. This partition reduction process may occur several times within a given recovery cycle and so it is not unexpected to see an increment of greater than 4096 listed for the recoveries statistic. XDR will only ship for the master partitions that it owns after the restart.
Solution
The most efficient and the recommended way to add a new DC is to add it dynamically. As a standard practice, when ever a dynamic change is done, we should also change the config file to reflect that change, so that if there is a restart in the future the same config takes effect.
Behaviour when adding a new DC dynamically.
When adding a new DC dynamically, please refer to the How To Enable XDR Dynamically In Aerospike 5.0 And Later article for detailed steps.
We only ship records from that point and not the whole namespace. If there is a need to ship from a certain point or the whole namespace, the How to rewind XDR for a namespace? feature can be leveraged.
NOTE: the latest versions from each lineage starting from 5.0 have the same behaviour. Only some intermediate builds from 5.1.0.3 had a different behaviour which was reverted as part of AER-6365.
For 5.1, it was fixed in 5.1.0.23 For 5.2, it was fixed in 5.2.0.15 For 5.3, it was fixed in 5.3.0.6.
Everything beyond 5.3.0.6 has a similar behaviour.
- Full documentation on how to configure XDR in both static and dynamic modes can be found here.