Problem Description
If a node is not performing as expected, for example slow disk, slow network behavior or CPU, it can appear to have a higher number of client connections compared to other nodes.Explanation
The slowness or increase in latency on that node may cause more number of requests to pile up on that node.Solution
Depending on the transaction throughput and the client side connection pool size, a temporary slow down of a node could end up causing the increase in the active client connections to persist. If the connections are never idle forproto-fd-idle-ms, the connections will simply be reused and having a higher count is not an issue on its own. One has to of course also monitor the proto-fd-max threshold which would prevent new connections from being established.Notes
How does the client policy (timeout and retries) affect this situation?
If the client times out or encounters socket errors and retries are configured, the client will close the socket and potentially open a new connection (to the same node or another node, depending on the transaction type and policy details) shortly afterwards on the subsequent attempts for the transaction. On the server, the connections may remain open for proto-fd-idle-ms (default 60 seconds). This will make the already potentially high number of client connections increase even further.