How to cleanly remove a namespace?

Audience: DBAs and SREs removing a namespace, or reusing the name of one they removed.
Applies to: Aerospike Database Enterprise Edition 7.1 and later. Data masking rules apply to 8.1.1 and later.

Summary

You remove a namespace by deleting its stanza from aerospike.conf and restarting each node. On restart, asd frees the namespace's shared memory. It does not erase two things:

• the records on the namespace's storage devices or files
• the cluster metadata (SMD) that names the namespace: its roster, secondary index definitions, truncate times, eviction cut-off, XDR filters and masking rules

While no node has the namespace, these leftovers do nothing. They take effect again if you add a namespace with the same name. For a strong-consistency (SC) namespace, the old roster comes back. Its partitions stay unavailable until every node in that old roster has the namespace. If its storage was wiped, every partition is then dead until you revive the namespace.

• If you will not reuse the namespace name, follow Remove a namespace you will not reuse.
• If you might reuse the name, follow Remove a namespace so its name can be reused.

What a restart cleans up

Why SMD items survive

• Nothing prunes them. SMD holds every item it was given and writes all of them to the module's .smd file. When a node starts without a namespace, each module skips that namespace's items, but SMD keeps them.
• Running nodes send them back. SMD keeps its items in sync across the cluster. A node that rejoins gets back every item it lacks from the nodes still running. During a rolling restart, deleting an item from the .smd file on the restarting node therefore has no effect. A running node also rewrites its .smd file from memory, so editing the file while asd runs does nothing.
• Wiping storage does not touch SMD. SMD lives in the work directory, not on the namespace's devices.

What comes back when the name is reused

Each SMD module stores its items under keys that begin with, or contain, the namespace name.

Every clearing command in this table works when no node has the namespace. Only the roster and the XDR shipping position need all nodes stopped.

Roles that hold privileges scoped to the namespace keep them. Privileges are scoped by namespace name, so they apply again to a namespace with the same name. Revoke them if the reused name must not inherit them.

To see what SMD holds for a namespace, run the following. Repeat it for each of roster, sindex, truncate, evict, XDR and masking:

asadm --enable -e "asinfo -v 'smd-show:module=roster'"

Each node returns a list of key=value items separated by ;, or <empty>. Look for keys that equal the namespace name, start with <ns>|, or contain |<ns> after a DC name.

Remove a namespace you will not reuse

1. Stop application traffic to the namespace.

2. If XDR ships the namespace, remove it from each DC:

   asadm --enable -e 'manage config xdr dc <dc> remove namespace <ns>'
   

   Also remove it from each DC's namespace stanza in aerospike.conf. asd does not start if a DC in aerospike.conf names a namespace that is not configured, and it logs DC <dc> has invalid namespace '<ns>'.

3. If security → log → report-data-op names the namespace, remove that line. asd does not start with a report-data-op that names a namespace that is not configured.
4. Remove the namespace stanza from aerospike.conf on every node, then restart the nodes one at a time. For an SC namespace, its partitions become unavailable as nodes leave it. This is expected, because the namespace is being removed.
5. Reclaim the namespace's storage. See Wipe the storage.

The SMD items left behind do nothing while no node has the namespace.

Remove a namespace so its name can be reused

Before you remove it

While the namespace is still configured:

1. List the SMD items that name the namespace, using smd-show as shown above.

2. Drop each secondary index:

   asadm --enable -e 'manage sindex delete <index-name> ns <ns>'
   

3. Clear each DC's XDR filter for the namespace:

   asadm --enable -e "asinfo -v 'xdr-set-filter:dc=<dc>;namespace=<ns>;exp=null'"
   

4. Remove each data masking rule on the namespace:

   asadm --enable -e 'manage masking drop namespace <ns> set <set> bin <bin> type <type>'
   

5. Revoke any role privileges scoped to the namespace that the reused name must not inherit.

Remove it

Follow steps 1–4 of Remove a namespace you will not reuse.

After no node has the namespace

1. Undo any truncates, and clear the eviction cut-off:

   asadm --enable -e 'manage truncate ns <ns> undo --no-warn'
   asadm --enable -e 'manage truncate ns <ns> set <set> undo --no-warn'
   asadm --enable -e "asinfo -v 'eviction-reset:namespace=<ns>'"
   

   Run the truncate undo only after no node has the namespace. An undo on a namespace that still holds data brings back truncated records that have not yet been removed from storage. With security enabled, run the truncate undo as a user whose truncate privilege is global, not scoped to the namespace. A privilege scoped to a namespace that no longer exists does not match.

2. Wipe the storage. See Wipe the storage.
3. Deal with the roster and the XDR shipping position. No command deletes them. Either:
   • leave them, and set a new roster and revive the namespace after you add it back (Reuse the name of a strong-consistency namespace), or
   • purge them with all nodes stopped (Purge SMD items with all nodes stopped).

asinfo commands run through asadm go to every node. The commands above change cluster-wide metadata, so each node repeating the change is harmless.

Wipe the storage

• Raw devices: zero the whole device, not just its header. A cold start reads past a wiped header and loads the old records it finds. On SSDs that return zeroes after a discard, blkdiscard is equivalent.
• Files: delete the namespace's storage files.
• Index mounts: if no remaining namespace uses an index-type or sindex-type flash or pmem mount, delete the files the namespace left on it while asd is stopped.

Purge SMD items with all nodes stopped

This is the only way to remove the roster and the XDR shipping position. It also removes the namespace's other SMD items, so you can use it instead of the clearing commands. Every node must be stopped at the same time. If any node is running, it sends the items back to the others when they start.

1. Stop asd on every node.
2. On every node, back up <work-directory>/smd/.

3. On every node, remove the namespace's items from each .smd file. Each file is a JSON array: the first element is a version pair, and each item after it is an object with a key. Keep the first element, and remove the objects whose key names the namespace. For example, with jq:

   cd /opt/aerospike/smd
   NS=<ns>
   # roster and evict: key is the namespace name
   for f in roster.smd evict.smd; do
     jq --arg ns "$NS" '[.[] | select(type != "object" or .key != $ns)]' "$f" > "$f.new" && mv "$f.new" "$f"
   done
   # sindex, truncate, masking: key is the namespace name or starts with "<ns>|"
   for f in sindex.smd truncate.smd masking.smd; do
     jq --arg ns "$NS" '[.[] | select(type != "object" or (.key != $ns and (.key | startswith($ns + "|") | not)))]' "$f" > "$f.new" && mv "$f.new" "$f"
   done
   # XDR: key is "<dc>|<ns>" or "<dc>|<ns>|F"
   jq --arg ns "$NS" '[.[] | select(type != "object" or ((.key | split("|"))[1] != $ns))]' XDR.smd > XDR.smd.new && mv XDR.smd.new XDR.smd
   

   Skip any file that does not exist. Check that each edited file is valid JSON (jq empty <file>) before you start asd. asd loads a file it cannot parse as empty, which drops every item in that module on that node, including those of other namespaces.

4. Start the nodes.

Reuse the name of a strong-consistency namespace

1. Add the namespace to aerospike.conf on every node, and restart the nodes one at a time.

2. When every node has the namespace, check its roster and partitions:

   asadm -e 'show roster'
   asadm -e 'show pmap'
   

   In show pmap, each node's row reports the cluster-wide Unavailable and Dead counts, so the total row multiplies them by the number of nodes.

3. If Current Roster for the namespace is empty (--) or differs from Observed Nodes, set the roster to the observed nodes and recluster. Copy the node IDs from Observed Nodes, separated by spaces instead of commas:

   asadm --enable -e 'manage roster stage nodes <node-id> <node-id> ns <ns> --no-warn'
   asadm --enable -e 'manage recluster'
   

   Use stage nodes with --no-warn when you run asadm with -e. manage roster stage observed asks for a confirmation code whenever the cluster is not stable, which it is not while the old roster lists absent nodes. With -e, that prompt fails and the roster does not change, but the manage recluster that follows still reports success. Run stage observed only in an interactive asadm session.

4. If Dead is above 0, revive the namespace and recluster. This happens when the old roster came back and the storage was wiped, or when the roster now lists nodes that have no data for the namespace. Setting a roster does not clear dead partitions.

   asadm --enable -e 'manage revive ns <ns>'
   asadm --enable -e 'manage recluster'
   

5. Confirm that Unavailable and Dead are both 0 in show pmap.