Detail
Background:
EBS volumes can be attached as persistent volumes (PV) to an EKS pod to use them as block devices in Aerospike. This can be configured like this:
PVs can be manually deleted by running commands like this:
Answer
Normally, we expect Aerospike to crash and produce errors in the logs when a configured device is removed. However, we can see in the screenshot below that the Aerospike process does not crash nor does it produce the expected errors after the PV has been deleted.
Kubernetes has a feature called Storage Object in Use Protection which actually prevents the deletion of objects that are actively in use. The purpose of the Storage Object in Use Protection feature is to ensure that PersistentVolumeClaims (PVCs) in active use by a Pod and PersistentVolume (PVs) that are bound to PVCs are not removed from the system, as this may result in data loss.
This means that if we delete an EBS volume from an EKS pod, the deletion of the volume will actually be postponed or delayed until it is not being used anymore. It is normal to not see errors in the Aerospike logs as technically the volume will still exist until it is not in use such as when the Aerospike pod is killed and/or recreated.