Detail
Q1: Can we upgrade from 2.1 to 3.2 be made as a single step or should i go through step jumps to perform this upgrade?A: It is possible to directly jump to latest 3.2.2 version provided that all the breaking changes made in 3.0 version are considered before upgrade. Also, all the migration steps mentioned in 3.0 docs should be followed.
AKO doesn’t support downgrade.
Ref: Upgrade operator 2.5.0 to 3.0.0 using Helm | Aerospike Documentation
Note: Direct upgrade to latest 3.2.2 is not possible in case of OLM.
Q2: The side effect of direct jump from 2.5 to 3.2?
A: No side-effects as such if all the required migration steps of 3.0 are followed. All the migration related code which is added in 3.0 is still present in 3.2. In case of any custom/forked Operator, it is advised to first test it in a testing environment.
Q3: The documentation talks about removing the v1beta1 reference post upgrade. Is this step optional? What is the side effect if we don’t run this?
A: AKO 3.0 introduced v1 APIs for the operator. It supports both v1 and v1beta1 APIs for functionality but K8s can store the object in ETCD with only version of the CRD. So as a result, when AKO is upgraded to 3.0, during initial reconciliation, it migrates all existing v1beta1 version objects to v1 version in the ETCD (via simple update call). And the step mentioned in the doc is to remove v1beta1 as the stored version after upgrade. These steps are as per k8s docs.
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#upgrade-existing-objects-to-a-new-stored-version
Q4: The documentation also states “Operator automatically migrates all existing Custom Resource objects to latest v1 version”
The customer wants to know how this works. If we could point them to the code then it’s also fine for them.
A. Introduction of new API and deprecation of old API in CRD require all existing old version objects stored in the ETCD to be migrated to new version (which is now the new storage version). This can be done manually, or with some storage version migration tool. In our case, this is done by AKO immediately after upgrade in the first reconciliation.
https://github.com/aerospike/aerospike-kubernetes-operator/blob/master/controllers/reconciler.go#L104C15-L104C38
Q5: Can we control this migration of v1beta1 and v1 objects or can this be disabled altogether.
A: No, this flow is not user-driver and can’t be controlled from outside.
Q6: Can the operator be upgraded using a canary deployment technique? Any recommendations here?
A: No, operator cannot be upgraded using a canary deployment.
Q7: If we achieve canary deployment, Is the Operator 3.2 capable of reconciling objects that are still on v1beta1?
A: Canary deployment is not supported.
Known Issue: When AKO is upgraded to 2.2.1 version, it results in rolling restarts of Aerospike cluster pods (due to webhook new FsGroup defaulting). This occurs when FsGroup in PodSepc security context is not set. If FsGroup is set to any value, this issue is bypassed.