superconfig/configs/monitoring/ at 0221fd0d5bd09ed57361585792fb104c4f4e1d70 · lass.pds.lassul.us/superconfig · Tangled

lass.pds.lassul.us / superconfig

0

This repository has no description

0

superconfig / configs / monitoring

1 folder 5 files

at 0221fd0d5bd09ed57361585792fb104c4f4e1d70

Hermes configs.monitoring: ignore NVMe endurance wear in SMART alerts 9d ago

configs.monitoring: export the superconfig revision each machine runs autoUpgrade pins the upgrade to the revision the forward-only guard resolved and writes it to /var/lib/superconfig-revision after a successful upgrade, like deploy does. Vector reads that file every 30s and exports nixos_system_info{host,revision,revision_short}; the node dashboard shows host -> revision_short.

1 week ago

configs.monitoring: alert on disk health and memory pressure smartctl_exporter runs on every non-container machine, bound to localhost and scraped by Vector into the host-tagged metrics path. Alerts on a failed SMART status, NVMe critical warnings, new NVMe media errors and NVMe spare below its threshold. Memory PSI comes from /proc/pressure/memory via an exec source, as a counter of stalled seconds, so the alert takes an exact 10m rate instead of the kernel's 5m avg300 EWMA. Plus an alert on any OOM kill. MemoryPressure threshold 10%: some task stalling on memory a tenth of the time over 10 minutes is sustained reclaim/swap thrash, while short spikes such as big builds average out.

1 week ago

configs.monitoring: ignore NVMe endurance wear in SMART alerts neoprism's NVMe disks report critical_warning 4 and a failing SMART status only because percentage_used is above 100% (rated endurance exceeded). Hetzner confirmed with the manufacturer that bit 0x04 is set for that reason and that the drives are fine as long as available_spare is above available_spare_threshold, which NvmeSpareLow watches. smartctl derives smart_status from the same bit, so both alerts stayed red on worn but healthy drives. NvmeCriticalWarning now fires only on critical_warning bits other than 0x04 (bitmap_and with 251). SmartFailed skips devices that smartctl_exporter reports with protocol="NVMe" and keeps covering SATA/SCSI disks.

1 week ago

configs.services.paint: add WBO whiteboard at paint.lassul.us

1 week ago

machines.orange: remove, move mumble-reminder to neoprism

1 week ago

retiolum: drop self.retiolum, acheron and starkstrom are in kartei now kartei carries both cards (krebs/kartei#92), so the superconfig-only host set is empty; read hosts, own addresses and tinc host files from kartei alone.

1 week ago