Why doesn't system memory usage go down after records expire or are deleted?

Detail

After a period of heavy expiration, eviction or deletes, the record count on a node drops and index_used_bytes falls, but the node's memory usage does not. system_free_mem_pct stays low, and ipcs -m still shows the same number of primary index shared memory segments as before. A normal (warm) restart of the Aerospike service does not change this. A cold restart does, as explained below.

This article explains how the primary index uses shared memory, why that memory is kept, why it is not a problem when capacity is planned for peak usage, and how to configure indexes-memory-budget so that a temporary peak can never leave a node stuck in stop-writes.

It applies to Enterprise Edition namespaces using the default index-type shmem.

Answer

How the primary index allocates memory

• Every record, including tombstones, uses one 64-byte entry in the primary index. This is counted for each copy, so the cluster-wide index size is 64 × replication-factor × records.
• The index is stored in Linux shared memory, in fixed-size blocks called arena stages. The stage size is set by index-stage-size (default 1 GiB). Each stage holds index-stage-size / 64 entries, which is 16,777,216 entries at the default.
• Stages are added one at a time, and only when the namespace has no free entries left. A namespace can have at most 2048 stages per node.
• Each namespace has its own arena on each node. Free entries in one namespace cannot be used by another namespace.

What happens when records expire

When a record expires, is evicted, or is deleted, its index entry is returned to the namespace's free list. The arena stage holding that entry stays allocated, and its memory stays resident. Aerospike does not release primary index arena stages back to the operating system while the process is running.

After a large expiration you will therefore see the following:

The gap between the allocated stages and index_used_bytes is free space inside the index. It is not a memory leak.

The freed space is reused

New records take entries from the free list before the index grows. After a large expiration, the namespace can grow back to its earlier peak record count without allocating any new shared memory. It only adds stages once the freed entries are used up.

In practice, a node's primary index shared memory reflects the highest record count that namespace has reached on that node since its last cold start, not its current record count. A node can reach a peak it would not otherwise reach when it takes on extra partitions while another node is out of the cluster.

Why a warm restart does not reclaim it, but a cold restart does

The shared memory segments are kept on purpose, because they allow a fast (warm) restart. A normal service restart or rolling restart is a warm restart: it reattaches to the existing segments, so the footprint is the same afterwards. Only a cold restart discards the segments and rebuilds the index from storage, sized for the current record count.

When this matters, and how to avoid problems

This behavior is not a problem when the node has capacity for its peak usage. It becomes a problem when a temporary spike grows the index close to the host's memory limit. After the spike expires, the memory stays allocated, so free system memory does not come back.

The namespace setting stop-writes-sys-memory-pct (default 90) stops writes when host memory usage goes above that percentage. Expiring or deleting records does not free host memory, so a system-memory stop-writes caused by index growth does not clear on its own, and a warm restart does not clear it either.

1. Plan capacity for the peak

Size each node's memory for its peak record count, not its average. The peak should include:

• Bulk loads, backfills and replays that are later removed by TTL or deletes.
• Tombstones from durable deletes, which use index entries until the tomb raider removes them.
• Extra partitions the node takes on when another node or rack is out of the cluster.
• Rounding up to a whole stage: a namespace that needs a little more than N stages allocates N+1.

See the capacity planning guide for the sizing formulas.

2. Set indexes-memory-budget for every namespace

indexes-memory-budget (Database 7.1 and later, dynamic) sets a per-namespace limit on the combined size of the primary, secondary and set indexes held in memory. When the indexes go over the budget, the namespace goes into stop-writes.

The budget is measured against the space the indexes actually use (for the primary index, records × 64 bytes), not against the shared memory allocated. So:

• A write spike is stopped at the budget, before it can grow the index to the host memory limit.
• When records expire or are deleted, used space drops back under the budget and the stop-writes clears automatically. No restart is needed.

With the budget set correctly, a spike causes a temporary, self-clearing index stop-writes instead of a system-memory stop-writes that only a cold restart can fix.

The budget is 0 (disabled) by default. Size it so that everything below fits under stop-writes-sys-memory-pct of host RAM:

sum over namespaces of (indexes-memory-budget
                        + one index-stage-size of rounding
                        + one sindex-stage-size of rounding, if the namespace has secondary indexes)
  + memory used by in-memory namespace data, if any
  + Aerospike process heap (heap_allocated_kbytes)
  + operating system and other processes
  <  stop-writes-sys-memory-pct  ×  host RAM

Example: on a 128 GiB host with one namespace, 90% is about 115 GiB. If the process heap and operating system need about 12 GiB, a budget of up to about 100 GiB keeps the index stop-writes ahead of the system-memory stop-writes, with room for one 1 GiB stage of rounding.

You can change the budget at runtime:

asadm -e "enable; manage config namespace <ns> param indexes-memory-budget to 100G"

Optionally, also set evict-indexes-memory-pct (default 0, disabled). This starts evicting records with a TTL when index usage reaches that percentage of the budget, before stop-writes is reached. Only use it if your application can accept records being evicted before their TTL.

3. Monitor used versus allocated

Compare what the index is using with what is allocated:

asadm -e "show statistics namespace like objects tombstones index_used_bytes indexes_memory_used_pct index_shmem"
asadm -e "show statistics service like system_free_mem_pct"
ipcs -m

On Database 8.2.0 and later, index_shmem_alloc_bytes reports the allocated primary index shared memory directly. On earlier versions, use ipcs -m.

In ipcs -m, the primary index segments are the ones whose size matches index-stage-size (1073741824 bytes at the default) and has a prefix of 0xae . The number of 0xae segments times the stage size is the allocated primary index shared memory for that node.

Reclaiming the memory

If a node needs its memory back (for example, it is already in a system-memory stop-writes after a spike), the only way to shrink the primary index is a cold restart. Do it one node at a time, following How to Recover Shared Memory (shmem) Usage with Rolling Cold Start.

Warning: A cold restart rebuilds the index from storage. Records that were deleted without durable deletes, or whose TTL was lowered, can come back. See Avoiding zombie records before cold restarting a node.

After the cold restart, set indexes-memory-budget as described above so that the next spike stops at the budget.

Notes

• This article covers the primary index. Secondary index stages (sindex-stage-size) are also kept after deletes and expiration. They are only released when every secondary index entry in the namespace is gone, for example when the last secondary index in the namespace is dropped.
• With index-type flash or pmem, the primary index does not use RAM, so it does not affect system memory or the index budget.
• Changing index-stage-size requires a cold restart. A smaller stage size (minimum 128 MiB) reduces the rounding overhead on small namespaces, but lowers the maximum number of records per node.
• Related articles:
  • Why is system_free_mem_pct low? (the same effect caused by partition rebalancing)
  • How do I identify Shared memory segments?
  • Does Shared Memory Get Fully Allocated Immediately
  • Inserts fail with arena allocation error on correctly sized all-flash cluster

Applies To Earliest Version

7.1.0 (for indexes-memory-budget; the arena behavior itself applies to all Enterprise Edition versions)

Applies To Latest Version

Current Version