Problem Description
When using the Distributed Mode of the bulk loader, sometimes we get error code 14 when writing vertices:
Caused by: com.aerospike.client.AerospikeException: Error 14,1,0,30000,1000,0,2 172.17.0.2 3100: Hot key
Error code 14 is a hotkey.
Explanation
When large Spark clusters spin up all their machines at once and a lot of Aerospike Graph Services start at the same time, they all request their id counter range. This could cause hotkeys when bulk loading.Solution
There are two things you could try as a workaround:
- Use smaller spark cluster
- Adjust transaction-pending-limit to a higher value. Please note that we do not recommend setting transaction-pending-limit to 0 as this could negatively impact your system. Before tuning transaction-pending-limit, you could also enable rw-client logging to get an idea of how many times a key is getting hit. For more details on enabling rw-client logging, please see our KB on How To use rw-client logging to identify a hot digest.
Notes
Impacts AGS 1.2.0 and below.Applies to Aerospike server versions 6.2 - current version