Aerospike Secret-Agent startup order

Problem Description

Aerospike Database is configured to read one or more values from Aerospike Secret Agent (secrets: in aerospike.conf), and both run as systemd services on the same host. After a host reboot, the Secret Agent service is running but the Aerospike service is not. Starting Aerospike by hand with systemctl start aerospike works every time.

The Aerospike log, or journalctl -u aerospike, shows messages similar to the following at the time of the reboot. The CRITICAL line names whichever setting was read first:

WARNING (secrets): failed to connect to secrets
CRITICAL (storage): {myns} can't get encryption key from secrets:MyResource:EncryptionKey
WARNING (as): SIGINT received, aborting Aerospike Enterprise Edition build 8.1.2.2 ...
WARNING (as): si_code SI_TKILL (-6)
WARNING (as): startup was not complete, exiting immediately

When the feature key comes from Secret Agent, the CRITICAL line reads CRITICAL (config): failed to get feature key secrets:... instead. When the agent connection uses TCP, Connection refused warnings for the agent's address and port may appear just before.

The systemd journal for that boot shows both services starting in the same second, with Aerospike exiting immediately and not starting again:

systemd[1]: Started aerospike-secret-agent.service - Aerospike Secret Agent.
systemd[1]: Started aerospike.service - Aerospike Server.
systemd[1]: aerospike.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: aerospike.service: Failed with result 'exit-code'.

The problem may come and go, and often appears for the first time after an Aerospike upgrade or an operating system patch.

Explanation

Aerospike asks Secret Agent for each secrets: value once, early in startup, and does not retry. If the agent is not accepting connections yet, Aerospike cannot load the value and stops startup. The SIGINT and si_code SI_TKILL lines are Aerospike shutting itself down cleanly after that error. They do not indicate a crash or an external kill.

At boot, systemd starts the two services in parallel. Neither of the systemd units shipped with Aerospike or Secret Agent orders one after the other. Aerospike reaches its first secret request within about a tenth of a second, while Secret Agent opens its listener a similar fraction of a second after it launches. Which one wins varies from host to host and from boot to boot. Small changes in startup speed, such as a new Aerospike version, a different package set or an OS update, can turn a host that always started cleanly into one that fails on every reboot.

The Aerospike unit file does not set Restart=, so systemd does not try again after Aerospike exits. systemctl enable aerospike makes the service start at boot. It does not make systemd restart it after a failure.

This affects every setting that accepts the secrets: prefix: feature-key-file, encryption-key-file, encryption-old-key-file, TLS cert-file, key-file and key-file-password, cert-blacklist, auth-password-file, default-password-file and query-user-password-file. It applies to TCP (secrets-address-port) and Unix domain socket (secrets-uds-path) connections alike.

Is this the problem you have?

Solution

Use a systemd drop-in for the Aerospike service so that it starts only after Secret Agent is accepting connections. Both options below change only systemd configuration; neither aerospike.conf nor the Secret Agent configuration needs to change. Apply the change to one node first and confirm it with a reboot before rolling it out.

What does not work on its own

• Ordering only. After=aerospike-secret-agent.service with Wants=aerospike-secret-agent.service and nothing else does not fix the problem. The Secret Agent unit is a simple service, so systemd considers it started as soon as its process launches, before it is listening. Aerospike still starts too early.
• Restart=on-failure with the default restart delay. systemd restarts the service after 100 ms and allows five starts in ten seconds. If Secret Agent takes more than about a second to become ready, for example while it initialises its connection to the secret manager, systemd uses up all five attempts and leaves Aerospike stopped with Start request repeated too quickly.

Option 1 (recommended): wait for Secret Agent before starting

Aerospike starts only once Secret Agent accepts a connection, so no failed start is logged.

Step 1. Create /usr/local/bin/wait-secret-agent, owned by root, with mode 755:

#!/bin/bash
# Wait until Aerospike Secret Agent accepts connections, then exit 0.
# Usage: wait-secret-agent <host> <port>     (TCP, e.g. 127.0.0.1 3005)
#        wait-secret-agent <socket-path>     (Unix domain socket, needs python3)
# Gives up after TIMEOUT seconds (default 60) and exits 1.
TIMEOUT=${TIMEOUT:-60}
probe() {
  if [ $# -eq 1 ]; then
    python3 -c "import socket,sys; s=socket.socket(socket.AF_UNIX); s.settimeout(1); s.connect(sys.argv[1])" "$1" 2>/dev/null
  else
    timeout 1 bash -c "exec 3<>/dev/tcp/$1/$2" 2>/dev/null
  fi
}
end=$((SECONDS + TIMEOUT))
while [ $SECONDS -lt $end ]; do
  probe "$@" && { echo "secret agent is accepting connections on $*"; exit 0; }
  sleep 0.5
done
echo "secret agent not accepting connections on $* after ${TIMEOUT}s"; exit 1

sudo chmod 755 /usr/local/bin/wait-secret-agent

The script makes a real connection, so it is not fooled by a stale socket file left behind by an unclean shutdown. It only opens and closes the connection and does not request a secret.

Step 2. Create the drop-in /etc/systemd/system/aerospike.service.d/secret-agent.conf. Use the same address and port as secrets-address-port in aerospike.conf:

[Unit]
Wants=aerospike-secret-agent.service
After=aerospike-secret-agent.service

[Service]
ExecStartPre=/usr/local/bin/wait-secret-agent 127.0.0.1 3005
Restart=on-failure
RestartSec=5

If Aerospike connects to Secret Agent over a Unix domain socket, pass the secrets-uds-path value instead:

ExecStartPre=/usr/local/bin/wait-secret-agent /path/to/agent.sock

Step 3. Reload systemd and check that the drop-in is applied:

sudo systemctl daemon-reload

systemctl cat aerospike

The output lists secret-agent.conf under the drop-in directory, followed by the lines above.

Step 4. Reboot the node, then confirm that Aerospike is running and that the wait step ran:

systemctl status aerospike

journalctl -b -u aerospike --no-pager | grep -E "secret agent|Started aerospike"

A line such as secret agent is accepting connections on 127.0.0.1 3005 appears just before Started aerospike.service.

Option 2: delayed restart, no script

If adding a script to the host is not an option, a delayed restart also brings Aerospike up reliably:

[Unit]
Wants=aerospike-secret-agent.service
After=aerospike-secret-agent.service

[Service]
Restart=on-failure
RestartSec=5

With this option the first start attempt can still fail. In that case the CRITICAL message above and a failed start appear in the logs on every reboot, and Aerospike comes up about five seconds later. If those messages feed alerting or audit processes, prefer Option 1.

Do I need Restart=on-failure?

The wait step in Option 1 is what prevents the boot race. Restart=on-failure covers two other situations:

The last row is a behaviour change. Some teams prefer a node that has stopped unexpectedly to stay down until someone has looked at it. If that is your policy, leave out Restart= and RestartSec=. The boot race is still fixed, but the two Secret Agent failure cases above then need a manual start.

Secret Agent on another host

When Secret Agent runs on a different host, omit the Wants= and After= lines, which only apply to a local service. Keep the ExecStartPre= line with the remote address and port, and keep Restart=on-failure with RestartSec=5.

Notes

• Aerospike documentation recommends starting Secret Agent before Aerospike Database: Install Secret Agent.
• Settings that accept the secrets: prefix: Integrating with secrets management services.
• This article covers hosts where both services are managed by systemd. Container and Kubernetes deployments start processes differently and are not covered here.

Applies To Earliest Version

Database 6.4 (Secret Agent support)

Applies To Latest Version

Current version (Database 8.2, Secret Agent 1.4.0)