Why is Aerospike Backup Service failing to read from S3 with a 403 error for the service account?

Detail

Why is authentication failing from the Aerospike Backup Service to AWS S3 storage from a newly configured service account.

The ABS logs shows the following error:

operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 403

Answer

When troubleshooting issues with  IRSA (IAM role for Service Account) , ensure the following are correct:

1. The backupService ServiceAccount is annotated with the correct IAM role arn to grant access.
   Example:
   

   kubectl annotate serviceaccount -n aerospike aerospike-backup-service eks.amazonaws.com/role-arn=arn:aws:iam::<ACCOUNTID>:role/<MyServiceRole>

2. The IAM Role trust policy is correct.

AKO docs for configuring S3:

https://aerospike.com/docs/kubernetes/tools/backup/backup-service/#:~:text=IAM%20Roles%20for%20Service%20Accounts%20(IRSA)

AWS guide for IRSA and trust policy:

https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html