Problem Description
A client (can also be asrestore) reports an error during operation:
Error while storing record - code 22: AEROSPIKE_ERR_FAIL_FORBIDDEN at src/main/aerospike/as_command.c:594Explanation
This error indicates that the record cannot be written. There can be different situations that cause this issue. The most common causes with solutions are covered below. Refer to the solution application to you.- Attempting to write to a set that is undergoing deletion.
- Attempting to write to a set that is under stop-write situation
- Attempting to write to a namespace via a regular client when it is configured to only accept writes from XDR client.
- Cluster clock skew between nodes is greater than 40 seconds.
- Attempting to write a record with TTL to a namespace which has namespace supervisor disabled.
Solution
-
Writing to a set that is being deleted (set-delete for server versions prior to 3.14) or a truncate command has been issued with a timestamp preventing writes. (Check for
set-deleteortruncatein the server logs).Solution: Avoid writing to set when a delete operation is ongoing. If the set-delete was in force during the write operation, wait for the set-delete to complete before attempting to write the records.
-
If a set reaches its
stop_writes. See the Managing Data in a Set documentation page for details.Solution: If
stop_writesis true, you will have to resolve either (depending on which threshold was reached) the lack of available disk % (min-avail-pct) or the memory usage (stop-writes-pct) before you can write any records.
-
If writing via a regular client when allow-nonxdr-writes is set to false or, for XDR traffic, when allow-xdr-writes is set to false. See fail_xdr_forbidden statistic for related information.
Solution: Verify and update the configurations to ensure they are set per your use-case.
-
As of Aerospike Server 4.5.1, for each Available mode (AP) namespace where nsup is enabled (i.e.
nsup-periodnot zero) writes will be suspended if cluster clock skew exceeds 40 seconds.Solution: Confirm if the cluster clock is skew within the specified limit
- If writing or updating records with a TTL while
nsup-periodis set to0. When using Aerospike Server 4.9 or above, the default value ofnsup-periodis set to zero (disabled). This means that the sub-system within Aerospike that controls eviction and expiration (the namespace supervisor) is not running by default and the database will not accept records with an expiration set.
Solution: The applicable solution depends on the desired behaviour of the database. If expiration and eviction are not required then when the record is written or updated do not set a TTL. This is done at an application level. If it is required to have records expiring or being evicted to free up space then enable nsup by changingnsup-periodto a value other than zero.If the
default-ttlis configured with a value other than0the database will not start withnsup-periodequal to zero.There is a parameter called
allow-ttl-without-nsupwhich allows the user to write records with a TTL even whennsup-periodis set to zero. The purpose of this parameter is to allow testing of the amount of CPU used by the namespace supervisor (NSUP) sub-system. This parameter should not be used unless it is part of a formalised testing process. Enablingallow-ttl-without-nsupis not recommended since records might that might be expected to expire will not be removed from the device or memory. For additional discussion, see Namespace Data Retention Configuration.