Context
Aerospike memory subsystem can be configured with allocation debugging.Memory debugging offers:
- Memory accounting: We keep track of which code location has allocated how much memory. This helps in the case of memory leaks.
- Double-free detection and corruption detection: This helps us find issues with Aerospike code, where there are potentially improperly managed allocations or some parts of the code making different assumptions about the allocations than other parts, causing unexpected bugs.
- There is a small memory overhead for keeping track of allocations, but this is almost always negligible (e.g. 4 additional bytes for a megabyte allocation). An exception may be data-in-memory namespaces, especially ones with smaller records. When using memory debugging in such a case, enable it on one node first to make sure the memory usage is still within bounds.
Method
By default, debug-allocations is disabled. Enabling this would assert on double frees and corruption detections which otherwise could go unnoticed. There is a potential risk when enabling this.
The configuration in the service stanza of the Aerospike configuration file and is a static configuration
(requires service restart).
service {
# ... previous configuration settings remain the same
debug-allocations transient
indent-allocations false
}
It takes one of the following four values as a parameter:
none- This completely disables any instrumentation of the allocation API: memory accounting, buffer overflow detection, double free detection. Instead, we simply forward any API calls directly to JEMalloc. In particular, this removes our 4-byte memory overhead per allocation.all- This enables instrumentation for all allocations and, thus, incurs the 4-byte memory overhead for all allocations.transient- This enables instrumentation only for transient - i.e., short-lived - allocations. Technically speaking, this exempts allocations from namespace arenas, whereas allocations from a thread’s default arena are covered by the instrumentation.persistent- This is the complement totransient. This setting enables instrumentation for allocations from namespace arenas, whereas allocations from a thread’s default arena are exempt.
Notes
These options do is they keep sanity checkingasd while it runs. When a memory inconsistency is detected, they’ll make asd crash with additional information that’ll hopefully allow us to track down the problem at hand. So, the options basically make asd more sensitive to any memory related bugs we might have in our code.Consequently, another downside is that these options can potentially cause
asd to crash more. Simply because it’s now more sensitive to memory-related bugs. So, that’s another reason to not aggressively deploy these options to a complete production cluster in one fell swoop. Start with one node and wait what happens. Otherwise, you risk that nodes are suddenly crashing left and right.Enabling
indent-allocations comes with a considerably higher memory overhead. When indent-allocations is enabled, the server will assert on detection of overwrites and all double frees. Also, each tracked allocation will incur a cost of 256 extra bytes.