Problem Description
The aerospike.log file is deleted to free space on a file system but the space is not released and when the lsof command is used to see what files are open, output similar to the following is produced.
asd 3004 root 4w REG 8,8 3120790837 78 /var/log/aerospike/aerospike.log-20190515 (deleted)
Explanation
This issue will occur when the aerospike.log file has been deleted while it is still in use by the Aerospike process (asd). When a file is opened the process that opens the file holds a file descriptor which is used to write to the file itself. The file will only be deleted when there are no processes with open file descriptors pointing to it. Storage space will not be reclaimed when an open file is deleted. When there are no further open file descriptors the operating system will reclaim the space.
Solution
If the log file has been deleted and the Aerospike service has not been restarted, then it may be possible to recover the log data using the following steps.
- Check
lsofto see if the asd process is still holding onto the file descriptor… - Run the following command
$ tail /proc/<pidof asd>/fd/*
This will show which file descriptor the asd process is writing to.
- Using the following file descriptor 5 and the process ID of the
asdis 7539 there will be a file called /proc/7539/fd/5. which represents the deleted file with the log data. - Copy the file to a backup directory.
$ cp /proc/7539/fd/5 /mnt/backup/aerospike.log.backup
- Issue a SIGHUP signal to the
asdprocess which will roll the log and start logging to a new file.
$ kill -HUP 7539
If it is not important to retain the log data the following command can be used to roll the logs to a new file. This will close the file descriptor and allow space to be reclaimed.
$ kill -HUP 7539