Problem Description
When there are a large number of backup files, precisely more than 1024, then the asrestore may fail with "Max key-put retries exceeded (5)" error.Explanation
The linux default per process file descriptors is 1024. Between opening more than 1024 backup files and a max of 32 * 16 async connections (each one a socket) (formula is max-async-batches * batch-size if server version < 6.0, default for max-async-batches is 32, batch-size default is 16 in this case) asrestore is exceeding the maximum open file descriptors. Maximum FD reached would cause the C client to fail to open new sockets, resulting in error -6 (AEROSPIKE_ERR_ASYNC_CONNECTION) which asrestore retries, resulting in the max-retry exceeded error. .Solution
It is recommended to check the per process file descriptor limit and raise it for asrestore if that is a problem, the appropriate number for pre 6.0 servers should benumber_of_backup_files + (max-async-batches * batch-size) For >= server 6.0 number_of_backup_files + max-async-batches. I’d give it a little over the result of that formula just to be safe. Or you can use fewer larger backup files to restore from.This article can be referred to increase the file descriptor limit for asd process.