Articles in this section

ABS full backup reports successful with missing namespaces after a retry attempt.

Problem description

A scheduled full backup in Aerospike Backup Service (ABS) completes and is reported as successful, but the backup run folder contains only some of the namespaces configured in the routine. For example, if a network error interrupts a backup partway through, ABS deletes the whole backup folder for that run, including namespaces that had already finished. It then retries only the namespace that failed, so the finished namespaces are never backed up again. The run is still reported as successful, even though some namespaces are missing. The end result in such scenario would be a backup routine with 6 namespaces produces a "successful" backup with only 4 or 2 namespace folders.

The issue is intermittent and is independent of the Aerospike Database server version. Also, it is seen only occurs on runs where more than one namespace are included, where at least one namespace backup fails and gets retried after other namespaces have already finished.

 

Typical symptoms:

  • The ABS log shows a transient connection error to one node during the scheduled run, followed by a retry, for example ResultCode: NETWORK_ERROR ... NO_AVAILABLE_CONNECTIONS_TO_NODE ... connection pool is empty.
  • The log shows Deleted folder path=<routine>/backup/<timestamp> at the same instant as Execution failed, retrying....
  • The run ends with set last backup time, and GET /v1/backups/currentBackup/<routine> reports the run as the last full backup.
  • Listing <routine>/backup/<timestamp>/data/ in the storage backend shows fewer namespace folders than the routine defines.
  • A manual (on-demand) full backup taken shortly afterwards contains all namespaces, because it does not hit a network error.

     

What is causing this issue?

This issue was identified a product bug in ABS (BKRS-405) versions till 3.6.2. When one namespace backup fails, its cleanup deletes the whole run folder that all namespaces share, not just its own namespace folder. 

Impacted versions - All ABS versions before 3.7.0 are affected

Workaround - A temporary workaround is to split the backup routine and let each namespace has its own routine.

 

Solution

Upgrade ABS to 3.7.0 or later. The fix is listed in the ABS 3.7.0 release notes under BKRS-405.

From 3.7.0, each namespace is retried on its own in a separate attempt folder under the run's timestamp (for example data/NAMESPACE.2). A failed attempt deletes nothing, and ABS removes earlier attempt folders only after the namespace completes. Because folder names can now carry an attempt suffix, check a run's contents with GET /v1/backups/full/<routine> or the run's metadata.yaml files rather than by listing folder names.

Before upgrading, do review the 3.7.0 breaking changes and follow Upgrade ABS guide.

 

References:

https://aerospike.com/docs/database/tools/backup-and-restore/backup-service/release/3-7-0/
https://aerospike.com/docs/database/tools/backup-and-restore/backup-service/upgrade/
https://aerospike.com/docs/kubernetes/reference/compatibility/#compatibility-matrix
 

Was this article helpful?
0 out of 0 found this helpful