Clone this repository
For self-hosted knots, clone URLs may differ based on your setup.
Since d9ed167 networkd installed the route to the container itself.
networkd adds it as soon as ve-riot gets carrier, before the container
start script runs `ip route add 10.233.1.2 dev ve-riot`, so the script
fails with "RTNETLINK answers: File exists" and container@riot ends up
in a restart loop. Before the reboot riot had never restarted since
d9ed167, so nobody noticed.
The script's route and address come from nixpkgs' nixos-containers
module and can't be turned off without patching nixpkgs (localAddress
also sets the container side), so the networkd route has to go. Now
ve-riot is Unmanaged=yes instead, and the start script owns its
address and route. Checked against systemd 261:
- An unmanaged link gets no configuration from networkd, so nothing is
in place before the script's `ip route add` and it succeeds on every
start.
- networkd only drops foreign routes for the link it is configuring
(link_drop_unmanaged_config, CONFIGURING state). When link_drop_routes
marks foreign routes it skips routes on other interfaces, "Otherwise,
routes assigned to unmanaged interfaces will be dropped". So a networkd
restart or reload no longer touches the script's route. That was the
bug d9ed167 fixed.
KeepConfiguration=static could not keep the script's route anyway: it
only protects foreign routes with protocol static, and `ip route add`
uses protocol boot. LinkLocalAddressing=no and KeepConfiguration=static
mean nothing on an unmanaged link, so they are gone, along with the
commented-out IPForward/IPMasquerade lines. Forwarding and NAT stay in
nftables and the sysctl. No other .network file on neoprism matches
ve-riot (99-ethernet-default-dhcp needs Kind=!*), but Unmanaged=yes
also keeps a later catch-all file from taking over ve-riot.
neoprism's NVMe disks report critical_warning 4 and a failing SMART
status only because percentage_used is above 100% (rated endurance
exceeded). Hetzner confirmed with the manufacturer that bit 0x04 is set
for that reason and that the drives are fine as long as available_spare
is above available_spare_threshold, which NvmeSpareLow watches. smartctl
derives smart_status from the same bit, so both alerts stayed red on
worn but healthy drives.
NvmeCriticalWarning now fires only on critical_warning bits other than
0x04 (bitmap_and with 251). SmartFailed skips devices that
smartctl_exporter reports with protocol="NVMe" and keeps covering
SATA/SCSI disks.
smartctl_exporter runs on every non-container machine, bound to
localhost and scraped by Vector into the host-tagged metrics path.
Alerts on a failed SMART status, NVMe critical warnings, new NVMe media
errors and NVMe spare below its threshold.
Memory PSI comes from /proc/pressure/memory via an exec source, as a
counter of stalled seconds, so the alert takes an exact 10m rate instead
of the kernel's 5m avg300 EWMA. Plus an alert on any OOM kill.
MemoryPressure threshold 10%: some task stalling on memory a tenth of
the time over 10 minutes is sustained reclaim/swap thrash, while short
spikes such as big builds average out.
flake.lock: update