Context
We used to enable rw-client log context to identify hotkeys for read and write transactions. Since server version 6.3, this is also possible to show digests for batch sub transactions in order to identify potential hotkey(s).Method
Since the introduction of server version 6.3, a new log context, "batch-sub," has been implemented to provide enhanced visibility into batch-sub-transaction activities. This log context includes the display of digests associated with each batch-sub-transaction.
To enable detailed logging for the batch-sub context for 10 seconds, execute the following commands:
asinfo -v "log-set:id=0;batch-sub=detail"
sleep 10
asinfo -v "log-set:id=0;batch-sub=info"
The resulting output in the Aerospike log will resemble the following examples:
DETAIL (batch-sub): (thr_tsvc.c:104) {test} digest 4491c5b2d5231a67192268f7cb21b2a4f51359e7 client 127.0.0.1:59592 write
DETAIL (batch-sub): (thr_tsvc.c:104) {test} digest c2489683af53f9373404e8c7e31742e705352133 client 127.0.0.1:59548 read
In the displayed log entries, aside from providing the digest, crucial information such as the client's IP address and the nature of the batch-sub transaction (e.g., read or write) is also included. This detailed logging capability enhances the monitoring and analysis of batch-sub-transaction activities, aiding in the identification and understanding of specific operations within the Aerospike database.
Here is the one line script to perform hotkey analysis:
grep -i detail /var/log/aerospike.log | awk '/digest/ {print $11}' | sort -k 1 | uniq -c | sort -k 1 | tail