Why do Queries fail on a cluster change?

Detail

During a cluster change (a node added, removed, restarted, or rejoining the cluster after maintenance), a query can exhaust all of its configured retries and fail.

Client logs show one sub-exception per failed attempt:

Error 11 ... Partition xxxx unavailable

Depending on the client version, the top-level exception carries either the last sub-error code (11, PARTITION_UNAVAILABLE) or MAX_RETRIES_EXCEEDED.

There are two distinct causes, and they need different fixes:

1. Tending time gap: the client's partition map is briefly out of date. This lasts about one client tend interval and is fixed with the query retry policy.
2. Partitions receiving migrations: a long query cannot reserve a partition that still has incoming migrations. This lasts until migrations for that partition complete, which can take minutes to hours. Retries do not help; it is fixed with the query duration policy.

Important (AP namespaces): Long queries are the default in every client, and a long query against an AP namespace requires each partition it reads to be full. As a result, long queries against AP namespaces fail with error 11 while migrations run after a node rejoins the cluster with data. Each application that queries an AP namespace must request a relaxed query, using LONG_RELAX_AP or SHORT, as described below.

Answer

How queries retry during cluster changes

As of Aerospike Database 6.0, clients issue queries against specific partitions. If some partitions are missed, only those partitions are retried, resuming from the last record received for each one. This allows partitions that are changing ownership ('in flight') during a cluster change to be retried so they are not missed.

A query fails only when a partition is still unavailable after the last retry.

Cause 1: Tending time gap

In most cases, a partition that is missed on one node is successfully retried against the next replica (with replication factor 2 or more). However, during migrations a node can temporarily claim ownership of more than one replica of a partition. The client learns partition ownership by tending each node in turn, so there can be a gap in which the client exhausts all of its retries before it has tended the rightful owner of a partition that was in flight.

Sample client log with a query policy of maxRetries=5, every attempt hitting node A1:

com.aerospike.client.AerospikeException:
Error 11,6,0,30000,0,5,A1 192.168.1.2 4333: Partition 1234 unavailable
sub-exceptions:
Error 11,1,A1 192.168.1.2 4333: Partition 1234 unavailable
Error 11,2,A1 192.168.1.2 4333: Partition 1234 unavailable
Error 11,3,A1 192.168.1.2 4333: Partition 1234 unavailable
Error 11,4,A1 192.168.1.2 4333: Partition 1234 unavailable
Error 11,5,A1 192.168.1.2 4333: Partition 1234 unavailable
Error 11,6,A1 192.168.1.2 4333: Partition 1234 unavailable

To cover this gap, adjust the query policy's maxRetries and sleepBetweenRetries so that the total retry window (maxRetries × sleepBetweenRetries) is comfortably longer than the client tend interval (tendInterval, 1 second by default).

Note: In the Java client, the query policy defaults are maxRetries=5 and sleepBetweenRetries=0. All retries can therefore complete within a few milliseconds, well inside a single tend interval. For example, maxRetries=5 with sleepBetweenRetries=500 gives a 2.5-second retry window.

Cause 2: Partitions receiving migrations

A query reserves each partition on the node it is sent to. How strict that reservation is depends on the query duration the client requests:

• A long query (the default) reserves a partition only if that node's copy is full, meaning it has no incoming migrations.
• A relaxed query reserves any copy of the partition that holds data, including one that is still receiving migrations.

When a node rejoins the cluster with data (for example after a rolling restart, upgrade, or OS patching), its data must be merged back. Until migrations for a partition complete, the node serving that partition has incoming migrations. A long query cannot reserve such a partition, and fails with Error 11 ... Partition xxxx unavailable on every retry until those migrations finish.

The failure window is the migration period after each node returns. Increasing retries does not help, because the partition stays unavailable for as long as its migrations run.

Note: A server older than 7.1.0 ignores LONG_RELAX_AP and runs the query as LONG.

What a relaxed query trades off

Queries do not reconcile record versions between copies of a partition. A relaxed query that reads a copy still receiving migrations returns what that copy holds at that moment. 

When the rolling maintenance procedure is followed, the copy a relaxed query reads already holds all recent writes, so a relaxed long query against an AP namespace does not miss records. See Queries: long queries.

Short queries against AP namespaces have always used relaxed reservation on the server versions listed above. LONG_RELAX_AP applies the same behavior to long queries.

Choosing a query duration

Small result sets (roughly 100 records per node or fewer)

Use SHORT. A short query also runs in a single thread and is subject to a server timeout (the client's timeout, or 1 second by default). It does not appear in the query monitor and cannot be aborted. It may run directly on a service thread when the namespace has inline-short-queries enabled. Do not use SHORT for queries that can return large result sets.

Larger result sets on an AP namespace

Use LONG_RELAX_AP. This requires Database 7.1.0 or later and a client version from the table below.

Results must come from full partitions

Keep LONG, and handle error 11 in the application: for example, re-run the query after migrations complete, or schedule such queries outside maintenance windows.

Note: When the namespace debug-only configuration force-long-queries is true, the server treats every query as long, including queries the client marks as SHORT.

Client support for LONG_RELAX_AP

The query duration is set per query policy, and the default in every client is LONG. The first client version that supports LONG_RELAX_AP is:

Earlier client versions offer only the boolean short-query option (for example, the Java client's QueryPolicy.shortQuery). On those versions, the only way to get relaxed reservation is to run the query as a short query, with the limits described above.

Notes

Reference:

• Queries: long queries
• Connection tuning guide: maxRetries and sleepBetweenRetries
• Database 7.1 enhancements: relaxed AP queries
• Using Aerospike policies correctly
• Client versioning and support
• Why do Queries fail when a node with data re-joins a cluster?
• Why am I getting unavailable partitions on scans in AP mode during migrations
• In Aerospike Database versions prior to 6.0, queries could miss a partition during cluster changes (when partition ownership changes because nodes are added or removed). That behavior was controlled by failOnClusterChange.

Applies To Earliest Version

6.0

Applies To Latest Version

Current Version