Detail
An Aerospike Client will open multiple connections to each node in an Aerospike cluster. What are these connections used for and how can the number of connections vary based on Client load and Server latency?Answer
When a client starts, it will be provided with the address of one node in the cluster to initially connect to. This is known as the seed address and can be a hostname or IP address, and you can even supply multiple addresses here, to enable an element of resiliency in the event of a node being down in the cluster.The client will take a single seed address and connect to it, and once connected it will ask the node to tell it about all of the other nodes in the cluster. The node will return a list of nodes in the cluster, along with their IP addresses and port numbers. The client will then connect out to every single node in the cluster. At this stage we have a single connection open to every node in the cluster.
Along these connections, each node will be asked to send us a list of which Aerospike partitions that node has. As each node returns it's list, the client will use this information to build a map of where all of the partitions live in the cluster, so that it knows where to go for each partition. This process is knowing as tending and is repeated down this same connection every 1 second (by default), which allows the client to notice changes in the Aerospike cluster. This connection is only ever used for tending and never for performing data transactions
Moving on to connections used to perform data transactions, the client will initially open minConnsPerNode connections to every node in the cluster. By default this is 0, so no additional connections will be opened. When a transaction comes in for a given node, the client will check whether it already has a connection open to that node, and if it does and it is not currently in use, it will use that connection. If there are no unused connections open to that node, then a new connection will be opened, subject to not exceeding maxConnsPerNode. If opening a new connection would exceed maxConnsPerNode, Error Code -7 "No More Connections" will be thrown.
Once the client has a connection for it's transaction (whether an existing connection which is unused, or by opening a new connection to the node), the transaction is sent down that connection. The client then waits for a response from the node down this connection, whether that be some data being returned (in the case of a read), or a confirmation of success (in the case of a write). The important point here is that this connection cannot be used for anything else during this time. It is dedicated to that single transaction.
In effect, what this means is that the client will open additional connections to a node in order to try to maintain the throughput being asked of it by the application. If an Aerospike node is struggling to keep up, this may result in additional connections being opened as each transaction is taking longer, resulting in connections being freed up less quickly and no longer being available for use by new transactions coming in to the client
The other thing to be aware of is that Aerospike supports the use of batch transactions. This is where transactions are batched together and then the client will send a smaller batch to each node with transactions destined for that node. In the explanation of connections above, a batch uses a single connection to each node. This means a batch is likely to be carry several individual transactions (as part of a batch) down a single connection. Only once all transactions in the batch for a given node have been completed will the connection carrying that batch transaction be freed up and available for reuse by another transaction