superconfig/configs/monitoring/ at 3a9014f48400a208b4a669fef56b6c9b9c8cbfc1 · lass.pds.lassul.us/superconfig · Tangled

lass.pds.lassul.us / superconfig

0

This repository has no description

0

superconfig / configs / monitoring

1 folder 5 files

at 3a9014f48400a208b4a669fef56b6c9b9c8cbfc1

Hermes configs.monitoring: match only systemd's own unit failure messages 5d ago

configs.monitoring: export the superconfig revision each machine runs autoUpgrade pins the upgrade to the revision the forward-only guard resolved and writes it to /var/lib/superconfig-revision after a successful upgrade, like deploy does. Vector reads that file every 30s and exports nixos_system_info{host,revision,revision_short}; the node dashboard shows host -> revision_short.

6 days ago

configs.monitoring: alert on disk health and memory pressure smartctl_exporter runs on every non-container machine, bound to localhost and scraped by Vector into the host-tagged metrics path. Alerts on a failed SMART status, NVMe critical warnings, new NVMe media errors and NVMe spare below its threshold. Memory PSI comes from /proc/pressure/memory via an exec source, as a counter of stalled seconds, so the alert takes an exact 10m rate instead of the kernel's 5m avg300 EWMA. Plus an alert on any OOM kill. MemoryPressure threshold 10%: some task stalling on memory a tenth of the time over 10 minutes is sustained reclaim/swap thrash, while short spikes such as big builds average out.

5 days ago

configs.monitoring: match only systemd's own unit failure messages SystemdUnitFailed matched any line containing "Failed with result", so a VictoriaLogs warning echoing a rejected query fired it for victorialogs.service. Anchor a regexp on _msg to systemd's "<unit>: Failed with result '<result>'." form.

5 days ago

configs.services.paint: add WBO whiteboard at paint.lassul.us

1 week ago

machines.orange: remove, move mumble-reminder to neoprism

6 days ago

retiolum: drop self.retiolum, acheron and starkstrom are in kartei now kartei carries both cards (krebs/kartei#92), so the superconfig-only host set is empty; read hosts, own addresses and tinc host files from kartei alone.

6 days ago