Problem Description
Kubernetes Operator throws an error “cannot update cluster security config enable-security was changed” while trying to enable security authentication after deployment.Explanation
The issue is mainly due to limitation mentioned in the document
-
Cluster security config flag "enable-security" cannot be updated after the first deployment
Solution
There is no direct method to surpass the limitation. But we have a workaround that should be tested in a non-production environment before any change is attempted in the production cluster.Steps:
-
Disable the Aerospike validation webhook. This bypasses the validation concerning the Aerospike security update. Use
kubectl get validatingwebhookconfigurations -n testto locate the Aerospike-related validation webhook. Change rules.resources fields from aerospikeclusters to some random string eg,aerospikeclusters-> aerospikeclusters123Use the command: kubectl edit validatingwebhookconfigurations aerospike-operator-validating-webhook-configuration
-
Add
security{}to theaerospikeConfigsection in thestatusof the AerospikeCluster CR by editing the CR.Use the command: kubectl edit aerospikeclusters <clusterName> --subresource=status -n <nsName>.
-
Add
security{}to theaerospikeConfigandaerospikeAccessControlsections in thespecof the AerospikeCluster CR by editing the CR.Use the command: kubectl edit aerospikeclusters <clusterName> -n <nsName>.
-
Connect to the cluster using asadm and assign
sys-adminanduser-adminroles to the admin user.kubectl exec <podName> -n <nsName> -it -- /bin/bash root@aerocluster-1-0:/# asadm -U admin -P admin Admin> enable Admin+> manage acl grant user admin roles sys-admin user-admin
-
Wait for the cluster to complete the warm restart of all the nodes. Please check for the
securityfeature if the same is enabled on all nodes using theAdmin> featurescommand in asadm. -
Re-enable the validation webhook by restoring the
resourcesfield or reversing the changes made in step 1.
FYI: When a single Operator is used to manage multiple clusters, changing the validation webhook could impact all other clusters. Consequently, it is recommended to create a new cluster and migrate the data for a safer and more isolated approach.