stockholm's kartei is a git submodule since bc08283, and github: flake
refs never fetch submodules, so evaluating #hotdog failed with
kartei/default.nix missing.

MCP server that controls the everyday Firefox through a WebExtension and
a native messaging host instead of Marionette/WebDriver, so
navigator.webdriver stays false and Cloudflare/reCAPTCHA stop flagging
normal browsing. No browser restart is needed to attach, tabs are
addressed by id on demand.

barnacle installs the server and links the native messaging host
manifest into /Library/Application Support/Mozilla/NativeMessagingHosts.
The extension is signed as unlisted on AMO via browser-mcp-sign.

atuin 18.23.0 fails this e2e test in unsandboxed darwin builds: the test
shell reads nix-darwin's /etc/bashrc, which resets PATH so the atuin
under test is not used. Fixed in nixpkgs by NixOS/nixpkgs#574509
(NOSYSBASHRC in preCheck); drop this package once our nixpkgs pin
includes it. Linux keeps the unmodified nixpkgs derivation.

ntfy.sh's free tier counts every UnifiedPush message against the
subscriber: 250 messages/day shared by all UnifiedPush apps on the phone.
Once used up it answers 429, phonetpm can no longer wake the phone and
every SSH login and deploy fails. Run our own ntfy on neoprism instead,
anonymous like ntfy.sh so app servers publish without accounts.

Rate limits: ntfy 2.29 defaults stay, except for an exemption.
- visitor-message-daily-limit is 0 (unlimited), only the request limiter
applies.
- visitor-subscriber-rate-limiting is off (ntfy.sh turns it on), so
publishes to up* topics are charged to the publisher, not to the phone.
The phone only spends requests on its subscription connections (30
allowed, the app uses one per server).
- The request limit is 60 burst, then one per 5s, per publisher IP.
Synapse on neoprism sends every Matrix push of the phone from one
address; encrypted and bridged rooms notify per message, so bridge
catch-up can exceed that. visitor-request-limit-exempt-hosts exempts
neoprism's own addresses. phonetpm daemons publish one wake per
connection and stay far below the limit.
- behind-proxy: ntfy takes the right-most X-Forwarded-For address, the
one nginx appends, so visitors are not all 127.0.0.1 and a client
cannot spoof its address.

The module enables attachments by default (attachment-cache-dir), set it
empty to keep them off. nginx settings follow ntfy's example config
(buffering off, 3m timeouts against the 45s keepalive, websockets).

No backup: the state dir (/var/lib/private/ntfy-sh) holds the message
cache (12h, disposable) and an auth db without users.

SystemdUnitFailed matched any line containing "Failed with result", so a
VictoriaLogs warning echoing a rejected query fired it for
victorialogs.service. Anchor a regexp on _msg to systemd's
"<unit>: Failed with result '<result>'." form.

Since d9ed167 networkd installed the route to the container itself.
networkd adds it as soon as ve-riot gets carrier, before the container
start script runs `ip route add 10.233.1.2 dev ve-riot`, so the script
fails with "RTNETLINK answers: File exists" and container@riot ends up
in a restart loop. Before the reboot riot had never restarted since
d9ed167, so nobody noticed.

The script's route and address come from nixpkgs' nixos-containers
module and can't be turned off without patching nixpkgs (localAddress
also sets the container side), so the networkd route has to go. Now
ve-riot is Unmanaged=yes instead, and the start script owns its
address and route. Checked against systemd 261:

- An unmanaged link gets no configuration from networkd, so nothing is
in place before the script's `ip route add` and it succeeds on every
start.
- networkd only drops foreign routes for the link it is configuring
(link_drop_unmanaged_config, CONFIGURING state). When link_drop_routes
marks foreign routes it skips routes on other interfaces, "Otherwise,
routes assigned to unmanaged interfaces will be dropped". So a networkd
restart or reload no longer touches the script's route. That was the
bug d9ed167 fixed.

KeepConfiguration=static could not keep the script's route anyway: it
only protects foreign routes with protocol static, and `ip route add`
uses protocol boot. LinkLocalAddressing=no and KeepConfiguration=static
mean nothing on an unmanaged link, so they are gone, along with the
commented-out IPForward/IPMasquerade lines. Forwarding and NAT stay in
nftables and the sysctl. No other .network file on neoprism matches
ve-riot (99-ethernet-default-dhcp needs Kind=!*), but Unmanaged=yes
also keeps a later catch-all file from taking over ve-riot.

neoprism's NVMe disks report critical_warning 4 and a failing SMART
status only because percentage_used is above 100% (rated endurance
exceeded). Hetzner confirmed with the manufacturer that bit 0x04 is set
for that reason and that the drives are fine as long as available_spare
is above available_spare_threshold, which NvmeSpareLow watches. smartctl
derives smart_status from the same bit, so both alerts stayed red on
worn but healthy drives.

NvmeCriticalWarning now fires only on critical_warning bits other than
0x04 (bitmap_and with 251). SmartFailed skips devices that
smartctl_exporter reports with protocol="NVMe" and keeps covering
SATA/SCSI disks.

smartctl_exporter runs on every non-container machine, bound to
localhost and scraped by Vector into the host-tagged metrics path.
Alerts on a failed SMART status, NVMe critical warnings, new NVMe media
errors and NVMe spare below its threshold.

Memory PSI comes from /proc/pressure/memory via an exec source, as a
counter of stalled seconds, so the alert takes an exact 10m rate instead
of the kernel's 5m avg300 EWMA. Plus an alert on any OOM kill.

MemoryPressure threshold 10%: some task stalling on memory a tenth of
the time over 10 minutes is sustained reclaim/swap thrash, while short
spikes such as big builds average out.