Problem Description
Why am I still seeing High water mark breached after clearing data and values for memory (and disk) are well below HWM? After clearing data, is it expected for HWM statistic to toggle to 'false' instantenously?
Admin> show stat like hwm
~~~~~~~~~test Namespace Statistics (2023-03-16 10:54:30 UTC)~~~~~~~~~~
Node |172.17.0.2:3000 |172.17.0.3:3000 |172.17.0.4:3000 |
hwm_breached|true |false |false |
Number of rows: 2
Explanation
Depending on the time of the HWM breach compared to when the namespace supervisor (NSUP) last ran, it can take up to whatever nsup-period is set to for it to be cleared.
For example, if you have nsup-period is set to 86400 (24 hours). If nsup just completed, it could take up to 24 hours for it to get cleared.
Solution
- Verify the last completion of Namespace Supervisor cycle and check if the nsup cycle is still running based on server logs
{ns-name} nsup-done: non-expirable 42162 expired (576066,922) evicted (24000935,259985) evict-ttl 134000 total-ms 155
- If nsup-period is set to a high value and you would like the HWM stat to stop showing hwm_breached true, you could try lowering the nsup-period to a lower value like 2-5 minutes instead.