How to gather server logs from every node with asadm collectlogs?

Audience: DBAs and SREs asked by Aerospike Support to send server logs from a cluster.

Applies to: asadm 2.22.0 and later (Aerospike Tools 10.2.1 and later). Behaviour described for asadm 5.1.0 (Tools 13.1.0).

Summary

collectlogs is an asadm command that gathers the Aerospike server log from every node in the cluster into one archive for Aerospike Support. It is separate from collectinfo, which does not include server logs. When Support asks for both, run both.

asadm -e "collectlogs --enable-ssh"

• asadm reads the log file from the node it runs on directly. It reaches every other node over SSH, and only when you pass --enable-ssh.
• For each node, asadm asks the server where it logs, then gzips each log on the node and copies it back over SFTP.
• The result is /tmp/collect_logs_<timestamp>.tgz on the host where asadm runs.

Most failures come from the SSH setup or from how the node logs. Check Before you run it first.

What gets collected

asadm collects each log sink the node reports, according to the sink type in the logging stanza:

A node with several sinks, for example both file and console, gets one file per sink.

The archive holds one file per node and sink, named <timestamp>_<node-id>_<log-name>.gz, plus <timestamp>_collectinfo_debug.log, which is asadm's own log of the run.

Console logging

When a node logs to console, the server has no log file to copy. asadm rebuilds the log from the systemd journal instead, so all of the following must hold:

• asd runs under the systemd unit named aerospike. That is the unit in the Aerospike packages. A unit with a different name, or asd started outside systemd, does not work:
  • If the journal has no aerospike unit entries from the last 24 hours, collection for that node fails.
  • If the journal still has entries from an earlier run of the aerospike unit, asadm collects those old lines and reports success. Nothing from the running process is included.
• Only the last 24 hours are collected. For an older incident, export the journal yourself (see Collect logs manually).
• Only lines that contain GMT are kept. Every server log line has a GMT timestamp, including with log-local-time true (for example GMT+0100). systemd's own lines about the unit, such as start and stop messages, are dropped.
• The SSH user must be able to read the system journal. Use root, or add the user to the systemd-journal or adm group. For any other user, journalctl returns no entries and collection fails.

For nodes in containers (Docker, Kubernetes) the console output goes to the container runtime, not to the host journal. Use docker logs or kubectl logs for those nodes.

Before you run it

Where to run asadm

You can run asadm on a cluster node or on a separate host that reaches every node on port 3000 and the SSH port.

• On a cluster node, asadm reads that node's log locally, with no SSH, as the local user. It recognises itself by matching the node's service address against the output of hostname -I. If the node advertises an address that is not on a local interface (for example, access-address set to a NAT or public IP), asadm treats it as remote and connects to it over SSH like any other node.
• asadm collects only the nodes it can see in the cluster. A node that is down or has left the cluster is skipped without an error.
• The archive and an uncompressed working copy are written under /tmp on the asadm host. Make sure /tmp has room for every node's compressed log. On each remote node, the compressed copy is written to /tmp/<timestamp>/<node-id>/ and removed after the copy. The empty /tmp/<timestamp> directory stays behind.

SSH requirements

asadm uses a built-in SSH client (the Python asyncssh library), not the ssh command. It connects to each node at the address asadm uses for that node, which is the address in the IP column of asadm -e "info network".

1. Host keys must already be trusted, by IP address.

asadm never prompts for an unknown host key and does not add new keys. Each node's host key must be in ~/.ssh/known_hosts of the user running asadm, under the node's IP address. An entry under a hostname does not match. StrictHostKeyChecking no and accept-new in ~/.ssh/config are ignored.

Add the keys from the asadm host:

ssh-keyscan -t ed25519,rsa,ecdsa 10.0.0.11 10.0.0.12 10.0.0.13 >> ~/.ssh/known_hosts

For a port other than 22, add -p <port>. The entries are then written as [10.0.0.11]:<port>, which is the form asadm needs. Check the fingerprints against your inventory if your security policy requires it. Connecting once with ssh <user>@<ip> and accepting the key does the same thing.

2. Authentication. Any of these methods works:

Command-line options override ~/.ssh/config. Without --ssh-user or a User entry, asadm logs in as the user running asadm. Without --ssh-port or a Port entry, it uses port 22.

3. The SSH user needs these permissions on each node. asadm runs plain commands, without sudo:

In practice, SSH as root is simplest. If root login is not allowed, give the account read access to the log as described above.

Check access before running

From the asadm host, test one node with the same user, key and port that asadm will use:

# file logging
ssh <user>@<node-ip> 'gzip -c /var/log/aerospike/aerospike.log | wc -c'

# console logging
ssh <user>@<node-ip> 'journalctl -u aerospike -a -o cat --since "1 day ago" | grep -c GMT'

The first command must print a byte count with no Permission denied error. The second must print a number greater than zero.

Run collectlogs

1. List the node addresses and confirm the cluster is complete:

   asadm -e "info network"
   

2. Make sure every address in the IP column is in known_hosts. See SSH requirements.

3. Run the collection:

   asadm -e "collectlogs --enable-ssh"
   

   Add --ssh-user, --ssh-key, --ssh-port or --ssh-pwd as needed. To name the output, add --output-prefix <prefix>, which gives /tmp/<prefix>_collect_logs_<timestamp>.tgz.

   If you authenticate with asadm's own login, keep those options in place, for example asadm -U <user> -P -e "collectlogs --enable-ssh".

4. Read the output before you send the archive. A clean run ends with:

   INFO: Successfully downloaded logs from all nodes.
   INFO: Files in /tmp/collect_logs_<timestamp> are now archived in /tmp/collect_logs_<timestamp>.tgz.
   

   Any ERROR: line names the node by (<ip>:<port>). The archive is still created with the logs that were collected. "Successfully downloaded logs from all nodes" does not prove that every node is included. It also appears when remote nodes were skipped because --enable-ssh was not given, and when a console-logging node's journal held only old entries.

5. Confirm that the archive has one file per node and sink:

   tar tzvf /tmp/collect_logs_<timestamp>.tgz
   

6. Upload the .tgz file to the support ticket. If any node is missing, see Troubleshooting or Collect logs manually.

Limits

• Each remote command has a 10-second timeout. This covers gzip of the log file and the journalctl export. A log too large to compress in 10 seconds fails for that node, typically a log larger than a few hundred megabytes to about 1 GB, depending on CPU speed. The error reason is empty, and the gzip process keeps running on the node until it finishes. For large logs, collect them manually. The local node has no such timeout.
• Only the active log file is collected. Rotated logs are not. If the incident happened before the last rotation, collect the rotated files manually.
• Console logs cover the last 24 hours only.
• syslog sinks are not collected. Collect the file that your syslog daemon writes, or export the journal by the syslog identifier (asd by default): journalctl -t asd.
• Two runs in the same second fail. The second run stops with [Errno 17] File exists: '/tmp/collect_info_<timestamp>'. Run it again.

Troubleshooting

Each entry starts with the message asadm prints, followed by its cause and fix.

Skipping downloading remote logs. SSH is not enabled.

• Cause: --enable-ssh was not given. Only the local node was collected.
• Fix: Add --enable-ssh.

Couldn't SSH login to remote server: Host key is not trusted for host <ip>

• Cause: The node IP, or [<ip>]:<port> for a non-default port, is not in ~/.ssh/known_hosts.
• Fix: Run ssh-keyscan for the node IP addresses. A hostname entry does not count.

Couldn't SSH login to remote server: Permission denied for user <user> on host <ip>

• Cause: Authentication failed: wrong user, no usable key, or wrong password.
• Fix: Check --ssh-user, --ssh-key, the agent, and authorized_keys on the node. Root login with a key needs PermitRootLogin prohibit-password or yes.

Couldn't SSH login to remote server: [Errno 111] Connect call failed ('<ip>', <port>)

• Cause: Nothing is listening on that SSH port.
• Fix: Check --ssh-port and any Port in ~/.ssh/config.

Passphrase must be specified to import encrypted private keys

• Cause: The key is encrypted. Nothing is collected.
• Fix: Add --ssh-key-pwd <passphrase>, or load the key with ssh-add.

Failed to compress <log path>: Process exited with non-zero exit status 1

• Cause: Usually the SSH user cannot read the log file. With a syslog sink the path is the syslog socket, which cannot be compressed.
• Fix: Use root, or grant read access. See SSH requirements. For syslog, collect manually.

Failed to compress <log path>: with nothing after the colon

• Cause: The 10-second timeout was reached.
• Fix: Collect that node's log manually.

Failed to copy journald to ...: Process exited with non-zero exit status 1

• Cause: Console logging, and journalctl -u aerospike returned no server lines from the last 24 hours. Either the SSH user cannot read the journal, or asd does not run under the aerospike unit.
• Fix: Use root or a systemd-journal or adm member. If asd does not run under the aerospike unit, collect manually.

Unable to download logs from node. This is likely an issue with your SFTP subsystem : Session request failed

• Cause: sshd on the node has no SFTP subsystem.
• Fix: Enable Subsystem sftp in sshd_config, or collect manually.

asadm's own debug log, <timestamp>_collectinfo_debug.log, is inside the archive and in the uncompressed /tmp/collect_logs_<timestamp>/ directory. It records which log paths each node reported.

Collect logs manually

When collectlogs cannot reach a node, or a log is too large, run these on each node and copy the results back with scp.

File logging, including rotated files:

tar czf /tmp/$(hostname)-aerospike-logs.tgz /var/log/aerospike/aerospike.log*

Console logging, for any time range:

journalctl -u aerospike -a -o cat --since "2026-11-01 00:00" --until "2026-11-03 00:00" | gzip > /tmp/$(hostname)-aerospike-console.log.gz

If asd runs under a different unit name, replace aerospike with that name. For syslog, use journalctl -t asd or the file your syslog daemon writes.