Detail
When a customer has configured records to have a TTL (time to live) and configured NSUP to run by setting nsup-period, when certain high water marks are reached, a process called eviction starts. Eviction is the forceable removal of records from the database. Here we discuss how it works.
Answer
When eviction is necessary, the records that are to be evicted are selected using an eviction histogram. The eviction histogram is comprised of a number of buckets that contain records. Each bucket is sized based on the TTL of the records.
The default number of buckets for the eviction algorithm is 10,000 although this can be configured between 100 and 10,000,000. The parameter controlling the number of eviction histogram buckets is evict-hist-buckets and it can be set as follows:
asinfo -v 'set-config:context=namespace;id=test;evict-hist-buckets=20000'
As implied by the use of asinfo, evict-hist-buckets is a dynamic parameter (can be modified without a cluster or node outage). If this value is modified it will come into effect in the next round of evictions. As shown in the asinfo command the context for the evict-hist-buckets parameter is the namespace so that it can be set on a per namespace basis. The command can also be included as a permanent setting within the aerospike.conf file.
The ability to configure the number of buckets used by the eviction algorithm addresses the issue of poor granularity within the TTL distribution as granularity can now be controlled. Thus, The width of each bucket in the expiration TTL histogram is still _maximum-record-ttl/100 while the width of each eviction histogram bucket is _maximum-record-ttl/evict-hist-buckets. This resolves issues of apparent uneven distribution and poor granularity within the TTL distribution when there is a large TTL variance within a single namespace. It follows then, that with the ability to manage the width of the eviction histogram buckets, it is easier to manage sizing, predict behaviour during evictions and fit TTL distribution more closely to the desired use case.
It should be noted that each eviction histogram bucket requires 4 bytes of memory and therefore if the maximum 10,000,000 buckets were set, the resultant eviction histogram would consume 40 MB of memory.
When evicting during a cold start process, the cluster will configure either 100,000 buckets or use the evict-hist-buckets, the highest value of the two will be chosen. This will only be valid for cold start eviction, when the cluster is running, evict-hist-buckets will be the dominant parameter.
Only entire buckets will be evicted. When a round of eviction occurs evict-tenths-pct is used to determine the number of records that must be evicted per NSUP cycle. This figure is used to define a threshold bucket. All buckets below (but not including) the threshold bucket will be evicted. A bucket is evicted completely or not at all.
Log Messages
When an eviction round is in progress the number of records to be evicted is shown clearly in an INFO level log message as follows:
Apr 07 2016 13:42:17 GMT: INFO (nsup): (thr_nsup.c:1079)
{test} found 12345 records eligible for eviction
The message above indicates that 12345 records are scheduled for eviction in the current round.
In some circumstances, no records will be evicted. This could be for a number of reasons and these will be shown in the logs.
Only records with a non-zero TTL will be eligible for eviction (meaning that records without a TTL or with a TTL of 0 will not be considered for eviction). If the namespace contains only records that cannot be evicted the following message will indicate this.
Apr 07 2016 13:42:17 GMT: WARNING (nsup): (thr_nsup.c:1065) {test}
no records eligible for eviction
As discussed earlier, only buckets below the threshold bucket will be considered for eviction. If the threshold bucket is the first bucket, this means that no buckets can be evicted. The following log message will indicate this (here 200346037 is the void time of the threshold bucket)
Apr 07 2016 13:42:17 GMT: WARNING (nsup): (thr_nsup.c:1068)
{test} no records below eviction void-time 200346037 - insufficient histogram resolution?
The message suggests that the eviction process could continue if the resolution of the histogram was increased (by setting a higher value for evict-hist-buckets) this will only work if the lower bucket contains a range of TTL. If the records in that bucket all have the same TTL they will remain in the same bucket regardless of resolution. Then increasing the evict-tenths-pct in intervals of 5 (as in 5, 10, 15, 20) will increase the total number of records to be evicted in that NSUP cycle and this increased number of to be evicted records may eventually become equal to the total number of records in the bucket and thus get evicted. Increasing the evict-hist-buckets may have a lesser impact than increasing evict_tenths_pct.
Jan 30 2017 02:36:21 GMT: WARNING (nsup): (thr_nsup.c:1043) {test} no records below eviction void-time 222541923 - threshold bucket 361, width 259 sec, count 686375 > target 530312 (0.5 pct)
The above log message suggests that no records below the indicated void time are eligible for eviction and thus we address the situation by either increasing the eviction histogram resolution(increasing evict-hist-buckets) and/ or by increasing the evict-tenths-pct periodically. Further, the log line indicates that the threshold bucket is 361 out of the number of evict-hist-buckets and 259 sec is the eviction histogram bucket width calculated as maximum-record-ttl/evict-hist-buckets. “count 686375 > target 530312” in the log line indicates that the eviction process found the first populated bucket to begin at that void time, and wanted to evict 530,312 records but found 686.375 records in that first bucket, thus no records are evicted, as partial buckets will not be evicted.
IMPORTANT: If NSUP is configured to evict 100% of the records or more in a single cycle (i.e.evict-tenths-pct set to 1000 or more), no records will be evicted and a message like the following will be logged:
Oct 24 2020 09:26:42 GMT: WARNING (nsup): (nsup.c:753) {ns_cache} would evict all 146897768 records eligible - not evicting!
Eviction of all evictable records can only be done manually, by setting the eviction depth far enough into the future that all records with TTLs expire. As the maximum TTL is ten years, setting the eviction depth ten years in the future should always suffice.