Problem Description
Migration got stuck while upgrading from CE to EE or adding new nodes to the cluster. Increasing the migrate-max-num-incoming and migrate-threads are not helpful.Aerospike logs show unexpected source with node id warnings as below
WARNING (migrate):(migrate.c:1619) ctrl ack (6): unexpected source 2ee273cxxxxxxxx
Explanation
By checking the logs, we can see two fabric address on the issue node, we could see the result as below.grep 'fabric published address' *
Sep 12 2023 18:06:08 GMT: INFO (fabric): (fabric.c:791) updated fabric published address list to {10.x.x.xx:13002,172.xx.xx.xxx:13002}
We could confirm the issue node has multiple interfaces configured, and Aerospike Fabric address binding are set to “any” (default value), during migrations fabric connections would span across these interfaces which cause migration stuck on an unexpected source. We recommend you to check fabric and heartbeat address bindings at server startup to avoid migration stuck or node may refuse to join the cluster.
Solution
Suggestion is to make fabric address binding consistent same as service and specify the IP address or interface name that the internal network should use for the binding :fabric {
address eth0 # specified the fabric address here or the interface 172.xxx.xxx.xxx
port 3001 # Intra-cluster communication port (migrates, replication, etc).
}
Similar issue may occur for heartbeat address also, please make heartbeat address binding with the hb service. Please note that rolling restart needed when binding fabric or heartbeat address.
More information on fabric address can be referred to:
https://docs.aerospike.com/reference/configuration/index.html#address