This repository has no description
configs.monitoring: ignore NVMe endurance wear in SMART alerts
neoprism's NVMe disks report critical_warning 4 and a failing SMART
status only because percentage_used is above 100% (rated endurance
exceeded). Hetzner confirmed with the manufacturer that bit 0x04 is set
for that reason and that the drives are fine as long as available_spare
is above available_spare_threshold, which NvmeSpareLow watches. smartctl
derives smart_status from the same bit, so both alerts stayed red on
worn but healthy drives.
NvmeCriticalWarning now fires only on critical_warning bits other than
0x04 (bitmap_and with 251). SmartFailed skips devices that
smartctl_exporter reports with protocol="NVMe" and keeps covering
SATA/SCSI disks.
author
Hermes
committer
lassulus
date
(Oct 16, 2026, 1:04 PM CEST)
commit
0221fd0d
0221fd0d5bd09ed57361585792fb104c4f4e1d70
parent
fe1f627a
fe1f627a751719bd4e2a05d8e33b83f882ce2f2f
+7
-2
+7
-2
configs/monitoring/alerts.nix
+7
-2
configs/monitoring/alerts.nix
···
52
52
(alert "OOMKill" "increase(host_memory_oom_kill_events_total[10m]) > 0" "0m"
53
53
"{{ $labels.host }}: {{ $value }} processes OOM-killed in the last 10 minutes"
54
54
)
55
-
(alert "SmartFailed" "smartctl_device_smart_status == 0" "5m"
55
+
# NVMe drives set critical_warning bit 0x04 (and so fail smart_status)
56
+
# once they exceed their rated endurance; their wear is covered by
57
+
# NvmeSpareLow instead. 251 masks out bit 0x04.
58
+
(alert "SmartFailed"
59
+
''smartctl_device_smart_status == 0 unless on (host, device) smartctl_device{protocol="NVMe"}''
60
+
"5m"
56
61
"{{ $labels.host }}: {{ $labels.device }} fails its SMART health check"
57
62
)
58
-
(alert "NvmeCriticalWarning" "smartctl_device_critical_warning != 0" "5m"
63
+
(alert "NvmeCriticalWarning" "bitmap_and(smartctl_device_critical_warning, 251) != 0" "5m"
59
64
"{{ $labels.host }}: {{ $labels.device }} reports critical warning {{ $value }}"
60
65
)
61
66
(alert "NvmeMediaErrors" "increase(smartctl_device_media_errors[1h]) > 0" "0m"