lass.pds.lassul.us / superconfig
This repository has no description
0
Fork 1

Configure Feed

Select the types of activity you want to include in your feed.

configs.monitoring: OOMKill ignores data gaps

When VictoriaMetrics has no sample shortly before the increase() window
(the monitoring server was down, or the series is new), MetricsQL's
increase() treats the first sample in the window as an increase from 0 and
returns the whole counter. After styx lost power for ~9 minutes, OOMKill
fired for coaxmetal with all 5 OOM kills since its boot 49 days ago,
although the counter never moved.

Only alert when the series has a sample in the minute before the window
(two minutes for the 60s smartctl scrape). Then increase() has a real
previous value, and still handles counter resets on a monitored host's
reboot. increase_pure() does not help: it assumes counters start at 0
and so counts the whole value as well.

NvmeMediaErrors uses increase() on a counter the same way and would
fire for any drive with past media errors after a gap, so it gets the
same guard. MemoryPressure's rate() is not affected: rate() does not
count the first sample in the window as an increase.

author
Hermes committer
lassulus date (Oct 19, 2026, 7:29 PM CEST) commit 2de6c647 parent 23ba6063
+9 -2
+9 -2
configs/monitoring/alerts.nix
··· 50 50 (alert "MemoryPressure" "rate(host_pressure_memory_some_seconds_total[10m]) > 0.1" "5m" 51 51 "{{ $labels.host }}: tasks stalled on memory {{ $value | humanizePercentage }} of the time" 52 52 ) 53 - (alert "OOMKill" "increase(host_memory_oom_kill_events_total[10m]) > 0" "0m" 53 + # Without a sample just before its window (a gap in VictoriaMetrics, 54 + # a new series) increase() returns the whole counter, so OOMKill and 55 + # NvmeMediaErrors require one via present_over_time. 56 + (alert "OOMKill" 57 + "increase(host_memory_oom_kill_events_total[10m]) > 0 and present_over_time(host_memory_oom_kill_events_total[1m] offset 10m)" 58 + "0m" 54 59 "{{ $labels.host }}: {{ $value }} processes OOM-killed in the last 10 minutes" 55 60 ) 56 61 # NVMe drives set critical_warning bit 0x04 (and so fail smart_status) ··· 64 69 (alert "NvmeCriticalWarning" "bitmap_and(smartctl_device_critical_warning, 251) != 0" "5m" 65 70 "{{ $labels.host }}: {{ $labels.device }} reports critical warning {{ $value }}" 66 71 ) 67 - (alert "NvmeMediaErrors" "increase(smartctl_device_media_errors[1h]) > 0" "0m" 72 + (alert "NvmeMediaErrors" 73 + "increase(smartctl_device_media_errors[1h]) > 0 and present_over_time(smartctl_device_media_errors[2m] offset 1h)" 74 + "0m" 68 75 "{{ $labels.host }}: {{ $labels.device }} had {{ $value }} new media errors in the last hour" 69 76 ) 70 77 (alert "NvmeSpareLow" "smartctl_device_available_spare < smartctl_device_available_spare_threshold"