Problem description
A scheduled full backup in Aerospike Backup Service (ABS) completes and is reported as successful, but the backup run folder contains only some of the namespaces configured in the routine. For example, if a network error interrupts a backup partway through, ABS deletes the whole backup folder for that run, including namespaces that had already finished. It then retries only the namespace that failed, so the finished namespaces are never backed up again. The run is still reported as successful, even though some namespaces are missing. The end result in such scenario would be a backup routine with 6 namespaces produces a "successful" backup with only 4 or 2 namespace folders.
The issue is intermittent and is independent of the Aerospike Database server version. Also, it is seen only occurs on runs where more than one namespace are included, where at least one namespace backup fails and gets retried after other namespaces have already finished.
Typical symptoms:
- The ABS log shows a transient connection error to one node during the scheduled run, followed by a retry, for example
ResultCode: NETWORK_ERROR ... NO_AVAILABLE_CONNECTIONS_TO_NODE ... connection pool is empty. - The log shows
Deleted folder path=<routine>/backup/<timestamp>at the same instant asExecution failed, retrying.... - The run ends with
set last backup time, andGET /v1/backups/currentBackup/<routine>reports the run as the last full backup. - Listing
<routine>/backup/<timestamp>/data/in the storage backend shows fewer namespace folders than the routine defines. -
A manual (on-demand) full backup taken shortly afterwards contains all namespaces, because it does not hit a network error.
What is causing this issue?
This issue was identified a product bug in ABS (BKRS-405) versions till 3.6.2. When one namespace backup fails, its cleanup deletes the whole run folder that all namespaces share, not just its own namespace folder.
Impacted versions - All ABS versions before 3.7.0 are affected
Workaround - A temporary workaround is to split the backup routine and let each namespace has its own routine.
Solution
Upgrade ABS to 3.7.0 or later. The fix is listed in the ABS 3.7.0 release notes under BKRS-405.
From 3.7.0, each namespace is retried on its own in a separate attempt folder under the run's timestamp (for example data/NAMESPACE.2). A failed attempt deletes nothing, and ABS removes earlier attempt folders only after the namespace completes. Because folder names can now carry an attempt suffix, check a run's contents with GET /v1/backups/full/<routine> or the run's metadata.yaml files rather than by listing folder names.
Before upgrading, do review the 3.7.0 breaking changes and follow Upgrade ABS guide.
References:
https://aerospike.com/docs/database/tools/backup-and-restore/backup-service/release/3-7-0/
https://aerospike.com/docs/database/tools/backup-and-restore/backup-service/upgrade/
https://aerospike.com/docs/kubernetes/reference/compatibility/#compatibility-matrix