To understand how a hot-key causes lag we need to understand some basic XDR concepts.
In XDR, each datacenter, namespace, and partition combination has its own distinct XDR transaction queue. A single DC thread for the configured source DC (XDR Data Center) iterates across these queues and hands work to the service thread pool. Although service-threads are configurable, all records belonging to the same partition are processed by the same service thread, so increasing service-threads does not parallelise a single hot partition.
Recovery processing uses configurable recovery threads. Recovery processing uses configurable recovery threads, while normal shipment processing consists of a single DC thread per configured XDR DC and the configurable service thread pool.
See our KB on XDR Recoveries HERE.
At a high level, when a client write occurs on the source DC (XDR Data Center), the write path records the record's digest and Last Update Time (LUT) and queues them into the appropriate XDR transaction queue. The DC thread later drains the queue and hands the digest to a service thread, which reads the record locally, prepares it for shipment, and sends it to the destination cluster.
For more information on the shipment lifecycle see our docs page HERE.
With the above workflow in mind, if a key is updated multiple times a second, the dc thread assigned to that partition becomes starved trying to process all these updates. This delays the shipment of the record and prevents the partition's Last Ship Time (LST) from advancing. Since namespace XDR lag is calculated using the oldest partition LST, the namespace lag starts to increase. This is because the XDR Lag metric is a calculation of the minimum LST of all partitions contained within that namespace, so even if only one partition in the namespace is slow it holds up the LST of the entire namespace, incrementing the XDR Lag counter.
In addition to the above behaviour, there is another situation where a hot-key can increment XDR Lag. XDR pipelines shipments across persistent network connections to the destination cluster. When hot keys are coupled with large record sizes, larger records take longer to serialise and transmit, which can occupy the shipment pipeline longer and delay subsequent shipments. This increases the partition's shipment delay and contributes to namespace XDR lag.
The fix:
hot-key-ms
With hot-key-ms set to 100ms (the default) XDR implements an internal suppression window of 100ms. If a duplicate key is found in the XDR transaction queue within the 100ms time specified of the last insert, the dc thread will not insert those duplicates into the transaction queue. This reduces duplicate queue entries for the same digest. If the same digest is already present in the transaction queue within the configured hot-key-ms window, XDR suppresses the duplicate queue insertion. This reduces unnecessary queue pressure while allowing the queued request to ship the latest record state available when it is processed.
In busy environments it is not uncommon to increase the hot-key-ms in excess of 1000ms. The maximum configurable value is 5000ms.
Reference Links:
- XDR record shipment lifecycle: https://aerospike.com/docs/database/manage/xdr/lifecycle#xdr-record-shipment-lifecycle
- XDR Recovery Mode: https://aerospike.com/docs/database/manage/xdr/lifecycle#xdr-recovery-mode
- Configuration Reference, hot-key-ms: https://aerospike.com/docs/database/reference/config#xdr__hot-key-ms