How to configure debug-allocations?

Lucien

Updated August 18, 2026 16:36

Context

The Aerospike memory allocation layer can be instrumented for debugging. When enabled, it provides:

Memory accounting — the server records how much memory each allocation site (code location) currently holds. This is the primary tool for investigating a suspected memory leak, and it is what Aerospike Support will normally ask for when process memory grows without a corresponding increase in data.
Corruption and double-free detection — each allocation carries a small trailer that is validated when the memory is freed. This catches double frees, buffer overwrites, and mismatched assumptions between different parts of the code that would otherwise go unnoticed.

Accounting adds 4 bytes of overhead per allocation. This is negligible for large allocations, but it is proportionally significant for very small ones — most notably in data-in-memory namespaces holding small records. Enable it on a single node first and confirm memory usage stays within expected bounds before extending it further.

Allocations made during the server's startup phase are exempt from instrumentation. Everything allocated after startup completes is covered.

Method

debug-allocations is set in the service stanza of the configuration file. It is a static parameter: it can only be applied at process start, and cannot be changed at runtime with asinfo set-config or asadm manage config. It is disabled by default.

The accepted values changed in server 7.0. Use the syntax matching your version.

Server 7.0.0.0 and later

The parameter is a boolean. The only accepted values are true and false.

service {
    # ... previous configuration settings remain the same
    debug-allocations true
}

Important: the values none, transient, persistent, and all were removed in 7.0. On 7.0 and later, any of them is a fatal configuration error, and the node will refuse to start, logging a message of the form:

CRITICAL (config): line 12 :: debug-allocations must be true or false or yes or no, not transient

If you are working from an older document or an older configuration file, check for this before restarting a node.

Server 6.4.x and earlier

The parameter takes one of four values, selecting which memory arenas are instrumented:

Value	Effect
`none`	Disables all instrumentation — accounting, overflow detection, and double-free detection. Allocation calls are forwarded directly to jemalloc, removing the 4-byte per-allocation overhead.
`transient`	Instruments short-lived allocations only. Allocations from a thread's default arena are covered; allocations from namespace arenas are exempt.
`persistent`	The complement of `transient`. Instruments allocations from namespace arenas; allocations from a thread's default arena are exempt.
`all`	Instruments all allocations, incurring the 4-byte overhead on each.

This arena-based distinction no longer exists in 7.0 and later, where instrumentation is all-or-nothing.

Verifying the setting

After restarting, confirm the setting is in effect:

asinfo -v 'get-config:context=service' | tr ';' '\n' | grep allocations

On 7.0 and later this reports debug-allocations=true or debug-allocations=false.

Collecting allocation data

Enabling the parameter only starts the accounting. To retrieve it, dump the jemalloc statistics and the per-site accounting with the jem-stats info command. If security is enabled, this requires a user with the logging-control privilege.

asinfo -v 'jem-stats:file=/tmp/jem.txt;sites=/tmp/sites.txt'

All parameters are optional:

file — destination for the jemalloc statistics. If omitted, the statistics are written to the Aerospike log instead.
sites — destination for the per-allocation-site accounting. This is the leak-hunting data, and it is only populated when debug-allocations is enabled; with the parameter off, the file will contain no accounting entries.
options — any combination of the characters gmablh, as defined by the jemalloc(3) man page. Only honoured when file is also specified.

Output paths are validated. In current 8.1.x releases the rules are:

An absolute path must be under /tmp/ or under the output subdirectory of the server's work directory. Anything else is rejected.
A relative path is resolved against that same output subdirectory.
The path must not contain .. anywhere, and must be under 100 characters.
The parent directory must already exist, must be writable by the server, and must not be reached through a symbolic link. The target file itself must not be a symbolic link.

Each run appends a new timestamped section rather than overwriting, so repeated captures accumulate in the same file.

A single snapshot only shows current totals. To identify what is growing, take at least three samples — shortly after the restart, midway through the workload, and again after memory has grown appreciably — and send all of them along with the server logs covering the same window.

Related parameters

These are also static parameters in the service stanza, and all are disabled by default. Enable them only when asked to, as each carries a higher cost than debug-allocations alone.

Parameter	Values	Effect and cost
`indent-allocations`	`true` / `false`	Offsets each allocation by a random amount so buffer overwrites are detected, and asserts on overwrites and all double frees. Adds 256 bytes per tracked allocation — a considerably higher memory overhead.
`poison-allocations`	`true` / `false`	Fills memory with a fixed byte pattern both when allocated and when freed, so reads of uninitialised or already-freed memory yield obviously invalid data. Applies to the first 64 MiB of an allocation.
`quarantine-allocations`	`0` to `100000000`	Withholds up to this many freed blocks from reuse and checksums them, crashing with a "use after free" message if a quarantined block is modified after being freed. Increases memory usage, since quarantined blocks are held out of circulation.

Notes

These options keep asd sanity-checking itself while it runs. When a memory inconsistency is detected, the server deliberately shuts down and records diagnostic details, rather than continuing in an uncertain state. The practical consequence is that a node running with allocation debugging is more likely to stop than one running without it — not because the options introduce faults, but because they surface latent ones that would otherwise pass silently.

For that reason, do not deploy these options across an entire production cluster at once. Enable them on one node, wait, and confirm the node is stable before extending to others. Otherwise you risk nodes stopping across the cluster at the same time.

Accounting tracks a maximum of 4096 distinct allocation sites. If the server exceeds this, it stops with a "too many call sites" message.

Allocation debugging is intended for diagnosis, not for steady-state operation. Once the data has been collected, remove the parameter and restart the node to return it to normal.

Applies To Earliest Version

Pre 4.9

Applies To Latest Version

Current Version