Troubleshooting ldap with ldapsearch

Audience: DBAs, platform engineers and LDAP / Active Directory administrators. Applies to: Aerospike Database Enterprise 8.1 and later with LDAP authentication.

Symptom

One of these:

• LDAP users can log in to Aerospike, but every data or admin command fails with error 81 (role violation).
• LDAP logins fail with error 93.
• A new AD group was added, but its members get no access.

Running Aerospike's own LDAP lookups with ldapsearch shows what the directory returns for each one, without restarting Aerospike. This article maps each ldap setting in aerospike.conf to its place on the ldapsearch command line.

1. What Aerospike does at an LDAP login

For each login that is not answered from cache (a successful login is reused for about 2 seconds, per user, per node), the node:

1. Binds as the query user (query-user-dn).
2. Finds the login user's entry with user-query-pattern under query-base-dn. The search must return exactly one entry, whose DN is the user's DN. If user-dn-pattern is set instead, the DN is built from the pattern and no search is made.
3. Binds as the login user with the password the client sent. This is the authentication.
4. On the same connection, now bound as the login user, runs each role-query-pattern under role-query-base-dn (or query-base-dn if that is not set). Aerospike asks only for the cn attribute of each entry found.
5. Compares each cn with the Aerospike role names, as an exact, case-sensitive match. Matches become the user's roles. Other entries are ignored silently.

A user with no matching role still logs in, and then gets error 81 on any command that needs a privilege.

Separately, the cluster's principal node re-runs steps 2 and 4 for every LDAP user every polling-period seconds, bound as the query user.

2. Where each setting goes on the ldapsearch command line

Settings with no ldapsearch equivalent: polling-period, session-ttl, login-threads, token-hash-method.

Which user goes where

3. Run the checks

Run these on an Aerospike node, so the network path and CA file are the same as the server's. -W prompts for the password of the -D account. -o ldif-wrap=no keeps long DNs on one line so they can be copied.

The examples use these values. Replace them with the ones from your aerospike.conf.

server                    ldaps://ldap1.example.com:636
tls-ca-file               /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
query-base-dn             dc=example,dc=com
query-user-dn             CN=svc-aerospike-query,OU=Service\20Accounts,DC=example,DC=com
user-query-pattern        (&(sAMAccountName=${un})(objectClass=person))
role-query-pattern        (&(objectClass=group)(member=${dn}))
login user                app-user

In aerospike.conf a space inside a value is written as \20, because the value ends at the first space. On the ldapsearch command line, use a real space inside single quotes.

If ldapsearch is not installed (RHEL / Rocky / Alma):

sudo dnf install -y openldap-clients

Point ldapsearch at the same CA file as tls-ca-file:

export LDAPTLS_CACERT=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

Step A: find the user's DN (bound as the query user)

Uses server, query-user-dn, query-base-dn and user-query-pattern with ${un} replaced:

ldapsearch -LLL -o ldif-wrap=no -H ldaps://ldap1.example.com:636 -D 'CN=svc-aerospike-query,OU=Service Accounts,DC=example,DC=com' -W -b 'dc=example,dc=com' '(&(sAMAccountName=app-user)(objectClass=person))' dn memberOf

Expect exactly one entry. Copy its dn: value for the next steps. memberOf: lists the groups the user belongs to directly.

Step B: check the login user's password (bound as the login user)

ldapsearch -LLL -o ldif-wrap=no -H ldaps://ldap1.example.com:636 -D '<user DN from step A>' -W -b '<user DN from step A>' -s base dn

Success means the bind Aerospike makes in step 3 of section 1 works. On failure, Active Directory's message includes a data code: 52e wrong password, 775 account locked, 532 password expired, 533 account disabled.

Step C: run the role query

Uses role-query-base-dn (or query-base-dn) and role-query-pattern, with ${dn} replaced by the DN from step A. Run it twice: once as the login user (what a login sees), once as the query user (what the periodic re-check sees).

As the login user:

ldapsearch -LLL -o ldif-wrap=no -H ldaps://ldap1.example.com:636 -D '<user DN from step A>' -W -b 'dc=example,dc=com' '(&(objectClass=group)(member=<user DN from step A>))' cn

As the query user:

ldapsearch -LLL -o ldif-wrap=no -H ldaps://ldap1.example.com:636 -D 'CN=svc-aerospike-query,OU=Service Accounts,DC=example,DC=com' -W -b 'dc=example,dc=com' '(&(objectClass=group)(member=<user DN from step A>))' cn

Every cn: returned is a candidate Aerospike role. Both runs should return the same list.

If the DN contains (, ), * or \, write them in the filter as \28, \29, \2a and \5c. Aerospike does this automatically when it substitutes ${dn}.

Step D: compare with the roles on the cluster

asadm <connection options> -U <admin-user> -P --enable -e "show roles"

A group grants access only if a role with exactly the same name, including case, exists on this cluster. If role-query-search-ou is true, the OU= values in the user's DN count as group names too.

4. Reading the results

5. Configuration-file rules that cause silent failures

1. One line, no quotes, no spaces. Each value ends at the first space, and quotes are kept as part of the value. Write spaces inside a value as \20.
2. Restart after a change. server, query-base-dn, role-query-base-dn, user-query-pattern, user-dn-pattern, role-query-pattern and role-query-search-ou are read only at startup. Restart the nodes one at a time. polling-period can be changed without a restart, by a user with the sys-admin privilege.
3. Identical on every node. Each login records the user's roles for the whole cluster, using the settings of the node that served that login.
4. Roles come only from LDAP. A role granted directly to an LDAP user in asadm is replaced at that user's next LDAP login.

6. Confirm on the server

Detailed security logging shows the filter each node actually sends. Turn it on with context security detail in a logging file section, or at runtime (it lasts until restart):

asadm <connection options> -U <admin-user> -P --enable -e "manage config logging file /var/log/aerospike/aerospike.log param security to detail"

Log in once as the test user, then:

grep -E "using the role query|found [0-9]+ roles|aerospike role|can't get roles|dn for user|error binding" /var/log/aerospike/aerospike.log | tail -20

With report-authentication true in the security log section, every login also writes an audit line. It ends detail: user=<user>;roles=<role1>,<role2>, or just detail: user=<user> when no roles were mapped.

Set the logging level back afterwards:

asadm <connection options> -U <admin-user> -P --enable -e "manage config logging file /var/log/aerospike/aerospike.log param security to info"

Reference: Configuring LDAP