Unexplained XDR Outbound Connector Connection Reset

Problem Description

XDR lag to a datacenter behind an Aerospike outbound connector (or the XDR Proxy) grows continuously and does not recover, while the destination system itself is healthy.

Typical symptoms, in combination:

• The XDR retry_conn_reset statistic for the affected DC and namespace climbs rapidly, often by orders of magnitude more than a comparable Aerospike-to-Aerospike DC on the same namespace.
• Aerospike server logs show repeated network timeout warnings for the connector endpoints:

WARNING (xdr-client): DC <dc-name> network timeout on <connector-ip>:<port>

• Connector logs show read timeouts and connection resets on the XDR-facing side:

ERROR metrics-ticker - ...netty.handler.timeout.ReadTimeoutException: count=<n>
ERROR metrics-ticker - ...NativeIoException - recvAddress(..) failed: Connection reset by peer: count=<n>

• XDR throughput collapses to a small fraction of normal, while the connector host itself shows low CPU, ample free heap, and a largely empty internal record queue.
• A packet capture on the connector host shows TCP ZeroWindow advertisements followed by RST, and netstat -s shows packets pruned from the receive queue because of socket buffer overrun.
• The problem appears only under sustained load, and may affect only some source nodes or only some connector instances rather than all of them.

Because the visible failures are TCP resets and timeouts, this is very commonly investigated first as a network or operating system fault. In the cases seen so far, the network and OS were found to be healthy.

Explanation

Outbound connector builds within a specific version range set a fixed 32 KB TCP receive buffer on every incoming XDR connection.

Before that change, the connector allowed the operating system to size the receive buffer, which lets Linux grow the TCP receive window under load (receive window auto-tuning). Setting the buffer size explicitly disables that auto-tuning, so every XDR connection is pinned at 32 KB — which the kernel presents as a 64 KB receive buffer, and which is typically below the system default.

The resulting sequence is:

1. Under sustained load the connector's fixed receive buffer fills faster than the connector drains it.
2. The connector advertises a TCP ZeroWindow to the Aerospike node, so no further data can be sent on that connection.
3. XDR applies a 10 second I/O timeout to in-flight records. This timeout is fixed in the server and is not configurable.
4. When the timeout expires, XDR logs the network timeout warning and resets the connection. Every record in flight on that connection is returned to the retry path.
5. XDR reconnects and the cycle repeats, producing a large and rapidly growing retry_conn_reset count and a backlog that cannot drain.

This is a throughput problem expressed as connection failures. The destination system — Kafka, JMS, Pulsar, or the destination Aerospike cluster — is not involved in the loop and is usually found to be healthy.

Note: although the original change was made to help deployments with large records and high throughput, connection resets have also been reproduced with small records. Record size alone should not be used to rule this out.

Affected and fixed versions

The fixed buffer was introduced by the Netty buffer customization change and reverted to operating-system sizing in the releases below. Every affected version in each product line is listed.

Note for XDR Proxy users: the change reached the XDR Proxy line one release later than the other connectors, in 3.2.4 and not 3.2.3. XDR Proxy 3.2.3 and earlier are not affected.

From the fixed versions onward, socket-receive-buffer-bytes has no default. It is applied only if you configure it explicitly, and the operating system otherwise sizes the buffer.

The same change was also released to the inbound connectors, which share the underlying library. Inbound connectors do not accept XDR connections, so the behaviour described in this article does not apply to them.

How to confirm

Run the following on a connector host while the problem is occurring.

1. Check the receive buffer size on the XDR-facing sockets

ss -tmin state established '( sport = :8080 )'

Use the port your connector listens on. Look at the skmem field on each socket:

skmem:(r66688,rb65536,t0,tb87040,...)

An rb value of 65536 on every socket, with no variation, is the signature of this issue.

For contrast, a healthy auto-tuned socket on a stock Linux host looks like this, with rb matching the system default and free to grow under load:

skmem:(r0,rb131072,t0,tb2626560,...)

The -n flag matters: without it, ss prints port 8080 as the service name webcache.

2. Compare against the system default

sysctl net.ipv4.tcp_rmem

The three values are minimum, default and maximum. Compare the middle value against the rb value seen in step 1. If the operating system default is larger than the observed rb, the buffer has been pinned by the application rather than by the operating system.

This difference is deterministic per connector version. Running an affected and a fixed build side by side on the same host, with the same configuration, produces:

affected build:  skmem:(r0,rb65536,...)
fixed build:     skmem:(r0,rb131072,...)     <- matches the OS default on that host

3. Check for receive queue pressure

netstat -s | grep -i -E "pruned|collapsed|listen queue|ZeroWindow"

Growing counts of packets pruned from the receive queue because of socket buffer overrun, and of the zero-window advertisement counters, support the diagnosis.

Note: netstat only prints counters that are non-zero, so empty output means no receive-queue pressure rather than a failed command. netstat also comes from the net-tools package, which is not installed by default on all RHEL 8 builds; install it first, or use nstat -az | grep -i -E "Prune|Collapse|ZeroWindow" instead.

4. Check the XDR side

asadm -e "show statistics xdr"

A large and rapidly increasing retry_conn_reset for the affected DC and namespace, especially when a second DC on the same namespace is healthy, confirms the impact.

Solution

Preferred: upgrade the connector to the fixed version for your product line from the table above, or later. No configuration change is then required, and the operating system resumes sizing the buffer.

Workaround without upgrading: set socket-receive-buffer-bytes explicitly in the service section of the connector configuration file.

service:
  port: 8080
  socket-receive-buffer-bytes: 131072

Guidance on the value:

• 65536 has been verified to stop the connection resets in a reproduction of this issue.
• 131072 is recommended where records are large or throughput is high. This is the value used in the example above.

Note that the kernel doubles whatever you configure, to allow for its own bookkeeping overhead, so 131072 produces a 256 KB receive buffer. That doubled figure is what ss reports as rb.

Before setting a larger value, check the system ceiling:

sysctl net.core.rmem_max

The kernel clamps your value to this limit before doubling it, so the effective result is min(configured, rmem_max) x 2. On a host where rmem_max is 212992, configuring a value above that is silently capped. Raise net.core.rmem_max first if you need a larger buffer.

If resets continue after raising the buffer, the following two settings have also been observed to help on affected builds:

service:
  max-concurrent-records: 65536
  enable-tcp-xdr-ack-buffering: false

A connector restart is required for any of these changes. Where multiple connector instances are deployed, restart them one at a time so that XDR can continue shipping through the remainder.

Notes

• This affects the shared service layer used by all Aerospike outbound connectors and the XDR Proxy, which is why the same fix was released across every product line on the same date.
• The 10 second XDR I/O timeout that triggers the connection reset is fixed in the server and cannot be tuned. Raising the receive buffer, or upgrading, is the only way to prevent the timeout being reached.
• Where a load balancer or ingress gateway sits between the Aerospike cluster and the connector, the symptoms have been observed to be more severe. If you have such a component in the path, include it in your investigation.
• Reference: socket-receive-buffer-bytes is documented in the service section of the outbound connector configuration.

Applies To Earliest Version

Aerospike Connect for Kafka - Outbound 5.3.3, JMS - Outbound 4.2.3, Pulsar - Outbound 3.3.3, ESP 2.4.3, Elasticsearch 2.2.3, XDR Proxy 3.2.4

Applies To Latest Version

Aerospike Connect for Kafka - Outbound 5.3.6, JMS - Outbound 4.2.6, Pulsar - Outbound 3.3.8, ESP 2.4.6, Elasticsearch 2.2.6, XDR Proxy 3.2.6