Commits
A machine job substituted the whole input closure of the derivations it
builds, e.g. 3.4 GiB for neoprism, of which the builds open a few dozen
MiB. .github/lazy-store.sh now runs before the build: it takes what
`nix build --dry-run` would fetch, collects its closure from the
narinfos of cache.nixos.org and cache.lassul.us (tools.lazy-store),
mounts lazy-store on /nix/store, registers the paths with
`nix-store --load-db` and copies their signatures. The build itself,
ci-build.sh's niks3 push and hestia stay as they were: outputs are
written to the real store dir below the mount, and ci-build.sh's
before/after diff of the store starts after the registration.
If anything fails before the paths are registered, it logs an error,
undoes the mount and exits 0, so the build substitutes eagerly. The FUSE
log and the fetched NARs are printed by an always-run step after the
build.
The eval job builds lazy-store next to niks3 and pushes it, the machine
jobs substitute it. The eval and shared jobs keep the plain store: eval
builds nothing but niks3 and lazy-store, and shared builds package
derivations, which read most of their inputs anyway.
FUSE file system for CI's machine builds, ported from the experiment on
ci-lazy-fuse (.github/exp/lazy-store). It mounts on /nix/store the real
store dir (all writes go there) merged with store paths of
cache.nixos.org and cache.lassul.us, whose metadata comes from their
.ls listings and whose NAR is fetched, checked and unpacked on the first
open() of a file in them; reads go through kernel FUSE passthrough. In
the experiment, rebuilding neoprism's system-units, etc, system-path and
toplevel downloaded 62 MiB instead of the 3,455 MiB of an eager
substitution.
Changes from the experiment:
- closure takes the store paths to make valid on stdin (what
`nix build --dry-run` would fetch) instead of walking derivations, and
leaves out paths whose closure is not entirely in the caches, so that
nix never sees a valid path with a missing reference; nix substitutes
those as usual.
- closure also writes the fetched narinfos as a file:// binary cache:
`nix-store --load-db` registers no signatures, and niks3's
--ignore-upstream-key and hestia's upstream-cache-filter go by the
signatures in the local database, so without them both would upload
(and thereby download) every cache.nixos.org path in an output's
closure. `nix store copy-sigs` adds them from there.
- mount fails if a path has no listing instead of leaving it out, as
every path gets registered.
Static, so that once it serves /nix/store it loads nothing from there.
A flake package, so CI builds it once and pushes it to cache.lassul.us,
and the jobs substitute it. passthru.tests.fetch covers fetch, closure
and the registration with signatures against a file:// cache.
The GIBS VIIRS_NOAA20_Thermal_Anomalies_375m_All raster shows each
detection as a coarse blob, far bigger than its 375 m footprint at
wallpaper scale, so the fire layer was loud and distracting.
Fetch FIRMS' keyless global 24 h VIIRS CSVs of NOAA-20, NOAA-21 and
Suomi NPP instead (cached, refreshed hourly like the old layer), drop
low-confidence detections and draw each one as an antialiased dot in
screen space through the same forward mapping as the markers: about
1 px at 1080p, growing slightly with the lens zoom, opacity from FRP on
a log scale between 1 and 500 MW. The colour and max blend under the
clouds stay as before.
Mercator at 16:9 only showed about +-71 degrees, cutting off northern
Greenland and the Arctic. The new variant world is plate carrée, north up,
centred on the equator and the home longitude, showing -90..90 degrees
stretched vertically to fill the screen. It replaces mercator in the CLI
--projection choices and the server's variant parameter, without an alias
for the old name. lens and plain are unchanged.
bluetoothd's hostname plugin overrode General.Name (and Class) with the
hostname and chassis, so disable it. bluetooth-pairing set adapter
properties before AutoEnable powered hci0, failed with Not Powered and
failed the switch; wait for Powered first.
acheron gets BlueZ (adapter "gg23", UB500 dongle) as an A2DP sink only;
WirePlumber's bluez monitor targets received streams at the "hub" sink,
like roc-lan and AirPlay. A bt-agent (NoInputNoOutput) accepts pairing
without PIN. A Home Assistant button (MQTT discovery) opens a 120s
discoverable/pairable window; afterwards bonded devices are trusted so they
reconnect any time, and a binary_sensor shows the window state.
ci.yml builds the machines with hestia now. Drops farm.yml, tools/ci-farm, neoprism's nix-grpc-daemon builder, barnacle's farm client, the nix-farm vars generators and the nix-grpc-store input. cache.lassul.us/niks3 moves to configs/nix-cache.nix.
Daily garbage collection of the hestia cache on the default branch, after
hestia's own gc.yml, so ci.yml's cache stays within the 10 GB GHA
cache quota.
Replaces the farm call. eval evaluates the new ciToplevels output once
with hestia's matrix subaction (the same machines the farm built: all
nixosConfigurations), shared builds the derivations needed by more than
one uncached machine once and hands the hestia manifest version with
their outputs to the machine jobs, one job per uncached machine builds
its toplevel by drv path. The ci job stays the required check; shared
and the machine jobs are skipped when nothing is uncached.
Every job pushes the store paths it added, built or substituted from
hestia, to niks3 with its GitHub OIDC token, as the farm did, so
deploys find the toplevels in cache.lassul.us. The niks3 client is the
new packages.niks3 output.
Measured on the ci-hestia-shared experiment with a nixpkgs bump: 15 min
(eval 3.3, shared 3.3, 1-4 per machine) instead of 49 with the farm. No
more queueing of runs for the farm.
levenshtein-merge only merged the managed CLAUDE.md for the wrapper.
coaxmetal's hermes keeps the plain llm.claude-code CLI.
stockholm's kartei is a git submodule since bc08283, and github: flake
refs never fetch submodules, so evaluating #hotdog failed with
kartei/default.nix missing.
MCP server that controls the everyday Firefox through a WebExtension and
a native messaging host instead of Marionette/WebDriver, so
navigator.webdriver stays false and Cloudflare/reCAPTCHA stop flagging
normal browsing. No browser restart is needed to attach, tabs are
addressed by id on demand.
barnacle installs the server and links the native messaging host
manifest into /Library/Application Support/Mozilla/NativeMessagingHosts.
The extension is signed as unlisted on AMO via browser-mcp-sign.
atuin 18.23.0 fails this e2e test in unsandboxed darwin builds: the test
shell reads nix-darwin's /etc/bashrc, which resets PATH so the atuin
under test is not used. Fixed in nixpkgs by NixOS/nixpkgs#574509
(NOSYSBASHRC in preCheck); drop this package once our nixpkgs pin
includes it. Linux keeps the unmodified nixpkgs derivation.
ntfy.sh's free tier counts every UnifiedPush message against the
subscriber: 250 messages/day shared by all UnifiedPush apps on the phone.
Once used up it answers 429, phonetpm can no longer wake the phone and
every SSH login and deploy fails. Run our own ntfy on neoprism instead,
anonymous like ntfy.sh so app servers publish without accounts.
Rate limits: ntfy 2.29 defaults stay, except for an exemption.
- visitor-message-daily-limit is 0 (unlimited), only the request limiter
applies.
- visitor-subscriber-rate-limiting is off (ntfy.sh turns it on), so
publishes to up* topics are charged to the publisher, not to the phone.
The phone only spends requests on its subscription connections (30
allowed, the app uses one per server).
- The request limit is 60 burst, then one per 5s, per publisher IP.
Synapse on neoprism sends every Matrix push of the phone from one
address; encrypted and bridged rooms notify per message, so bridge
catch-up can exceed that. visitor-request-limit-exempt-hosts exempts
neoprism's own addresses. phonetpm daemons publish one wake per
connection and stay far below the limit.
- behind-proxy: ntfy takes the right-most X-Forwarded-For address, the
one nginx appends, so visitors are not all 127.0.0.1 and a client
cannot spoof its address.
The module enables attachments by default (attachment-cache-dir), set it
empty to keep them off. nginx settings follow ntfy's example config
(buffering off, 3m timeouts against the 45s keepalive, websockets).
No backup: the state dir (/var/lib/private/ntfy-sh) holds the message
cache (12h, disposable) and an auth db without users.
SystemdUnitFailed matched any line containing "Failed with result", so a
VictoriaLogs warning echoing a rejected query fired it for
victorialogs.service. Anchor a regexp on _msg to systemd's
"<unit>: Failed with result '<result>'." form.
Since d9ed167 networkd installed the route to the container itself.
networkd adds it as soon as ve-riot gets carrier, before the container
start script runs `ip route add 10.233.1.2 dev ve-riot`, so the script
fails with "RTNETLINK answers: File exists" and container@riot ends up
in a restart loop. Before the reboot riot had never restarted since
d9ed167, so nobody noticed.
The script's route and address come from nixpkgs' nixos-containers
module and can't be turned off without patching nixpkgs (localAddress
also sets the container side), so the networkd route has to go. Now
ve-riot is Unmanaged=yes instead, and the start script owns its
address and route. Checked against systemd 261:
- An unmanaged link gets no configuration from networkd, so nothing is
in place before the script's `ip route add` and it succeeds on every
start.
- networkd only drops foreign routes for the link it is configuring
(link_drop_unmanaged_config, CONFIGURING state). When link_drop_routes
marks foreign routes it skips routes on other interfaces, "Otherwise,
routes assigned to unmanaged interfaces will be dropped". So a networkd
restart or reload no longer touches the script's route. That was the
bug d9ed167 fixed.
KeepConfiguration=static could not keep the script's route anyway: it
only protects foreign routes with protocol static, and `ip route add`
uses protocol boot. LinkLocalAddressing=no and KeepConfiguration=static
mean nothing on an unmanaged link, so they are gone, along with the
commented-out IPForward/IPMasquerade lines. Forwarding and NAT stay in
nftables and the sysctl. No other .network file on neoprism matches
ve-riot (99-ethernet-default-dhcp needs Kind=!*), but Unmanaged=yes
also keeps a later catch-all file from taking over ve-riot.
neoprism's NVMe disks report critical_warning 4 and a failing SMART
status only because percentage_used is above 100% (rated endurance
exceeded). Hetzner confirmed with the manufacturer that bit 0x04 is set
for that reason and that the drives are fine as long as available_spare
is above available_spare_threshold, which NvmeSpareLow watches. smartctl
derives smart_status from the same bit, so both alerts stayed red on
worn but healthy drives.
NvmeCriticalWarning now fires only on critical_warning bits other than
0x04 (bitmap_and with 251). SmartFailed skips devices that
smartctl_exporter reports with protocol="NVMe" and keeps covering
SATA/SCSI disks.
smartctl_exporter runs on every non-container machine, bound to
localhost and scraped by Vector into the host-tagged metrics path.
Alerts on a failed SMART status, NVMe critical warnings, new NVMe media
errors and NVMe spare below its threshold.
Memory PSI comes from /proc/pressure/memory via an exec source, as a
counter of stalled seconds, so the alert takes an exact 10m rate instead
of the kernel's 5m avg300 EWMA. Plus an alert on any OOM kill.
MemoryPressure threshold 10%: some task stalling on memory a tenth of
the time over 10 minutes is sustained reclaim/swap thrash, while short
spikes such as big builds average out.
autoUpgrade pins the upgrade to the revision the forward-only guard resolved and writes it to /var/lib/superconfig-revision after a successful upgrade, like deploy does. Vector reads that file every 30s and exports nixos_system_info{host,revision,revision_short}; the node dashboard shows host -> revision_short.
The flake's git revision (-dirty with uncommitted changes, unknown for non-git sources) is copied to the target with the source and written by deploy-switch once the switch succeeded. Read by the monitoring agent; it is in no derivation, so a commit doesn't change every machine's closure.
kartei carries both cards (krebs/kartei#92), so the superconfig-only host
set is empty; read hosts, own addresses and tinc host files from kartei
alone.
Query field values for host and unit (unit narrowed by the selected hosts), with multi-select and All = *, and filter the panel with in().
debugLevel 0 still maps to Info, so reconnect, MTU and path-revalidation messages flood journald. RUST_LOG has to be target-scoped: -d 0 sets the global directive and replaces a bare RUST_LOG=warn.
HTTP service rendering wallpapers on demand for a location, variant,
overlay and size. Renders stay fresh for 10 minutes; ones that were
picked up are re-rendered in the background when they go stale.
realwallpaper gains --cache-dir so locations share one data cache, and
its cache cleanup now tracks last use instead of fetch time.
Replaces pkgs.realwallpaper (xplanet, dead NASA NEO sources) with a
Python renderer: oblique cylindrical map centred on home with a zoom
lens (or plain/mercator), multi-resolution WMS imagery (Sentinel-2
cloudless, Blue Marble, Black Marble lights), live EUMETSAT clouds with
synthesised detail, VIIRS snow/fire/chlorophyll, sea ice and the sky.
The krebs.realwallpaper module, its nginx config and the mors
fetchWallpaper config are dropped; the tool runs standalone for now.
pkgs.phonetpm: 0.4.1 -> 0.4.2
Branches, PRs and auto-merge now use the UPDATE_PKGS_TOKEN repo secret
instead of GITHUB_TOKEN, so the normal pull_request ci.yml run (with the
farm build) shows up on each PR and GitHub merges it with a merge commit
once the ruleset is met. The serial verify job (rebase, dispatch ci.yml,
wait, merge) and the final ci dispatch on master are gone: the auto-merge
counts as the token's, so the push to master triggers ci.yml itself.
UPDATE_PKGS_TOKEN: fine-grained PAT (or GitHub App token) for this repo
with Contents: read/write, Pull requests: read/write and Workflows:
read/write (force-pushing an update branch rebased over master commits that
touched .github/workflows is refused without it). The repo needs "Allow
auto-merge" enabled. The workflow fails early if the secret is missing.
Grafana 13 no longer ships the prometheus datasource as a core plugin, so the provisioned VictoriaMetrics datasource was unusable. Switch it and the dashboards to victoriametrics-metrics-datasource.
Every NixOS machine runs Vector (host metrics, internal metrics, journal)
and writes to VictoriaMetrics/VictoriaLogs on both styx and coaxmetal,
each sink with a 512 MiB disk buffer. Both servers run vmalert (metrics and
LogsQL instances), a two-node Alertmanager cluster posting to Matrix via
matrix-alertmanager-receiver, Gatus (results scraped into VictoriaMetrics,
alerted on by vmalert) and Grafana with provisioned datasources and
dashboards. All ports are open on retiolum only.
Replaces telegraf on neoprism, including the borgbackup status file.
A machine job substituted the whole input closure of the derivations it
builds, e.g. 3.4 GiB for neoprism, of which the builds open a few dozen
MiB. .github/lazy-store.sh now runs before the build: it takes what
`nix build --dry-run` would fetch, collects its closure from the
narinfos of cache.nixos.org and cache.lassul.us (tools.lazy-store),
mounts lazy-store on /nix/store, registers the paths with
`nix-store --load-db` and copies their signatures. The build itself,
ci-build.sh's niks3 push and hestia stay as they were: outputs are
written to the real store dir below the mount, and ci-build.sh's
before/after diff of the store starts after the registration.
If anything fails before the paths are registered, it logs an error,
undoes the mount and exits 0, so the build substitutes eagerly. The FUSE
log and the fetched NARs are printed by an always-run step after the
build.
The eval job builds lazy-store next to niks3 and pushes it, the machine
jobs substitute it. The eval and shared jobs keep the plain store: eval
builds nothing but niks3 and lazy-store, and shared builds package
derivations, which read most of their inputs anyway.
FUSE file system for CI's machine builds, ported from the experiment on
ci-lazy-fuse (.github/exp/lazy-store). It mounts on /nix/store the real
store dir (all writes go there) merged with store paths of
cache.nixos.org and cache.lassul.us, whose metadata comes from their
.ls listings and whose NAR is fetched, checked and unpacked on the first
open() of a file in them; reads go through kernel FUSE passthrough. In
the experiment, rebuilding neoprism's system-units, etc, system-path and
toplevel downloaded 62 MiB instead of the 3,455 MiB of an eager
substitution.
Changes from the experiment:
- closure takes the store paths to make valid on stdin (what
`nix build --dry-run` would fetch) instead of walking derivations, and
leaves out paths whose closure is not entirely in the caches, so that
nix never sees a valid path with a missing reference; nix substitutes
those as usual.
- closure also writes the fetched narinfos as a file:// binary cache:
`nix-store --load-db` registers no signatures, and niks3's
--ignore-upstream-key and hestia's upstream-cache-filter go by the
signatures in the local database, so without them both would upload
(and thereby download) every cache.nixos.org path in an output's
closure. `nix store copy-sigs` adds them from there.
- mount fails if a path has no listing instead of leaving it out, as
every path gets registered.
Static, so that once it serves /nix/store it loads nothing from there.
A flake package, so CI builds it once and pushes it to cache.lassul.us,
and the jobs substitute it. passthru.tests.fetch covers fetch, closure
and the registration with signatures against a file:// cache.
The GIBS VIIRS_NOAA20_Thermal_Anomalies_375m_All raster shows each
detection as a coarse blob, far bigger than its 375 m footprint at
wallpaper scale, so the fire layer was loud and distracting.
Fetch FIRMS' keyless global 24 h VIIRS CSVs of NOAA-20, NOAA-21 and
Suomi NPP instead (cached, refreshed hourly like the old layer), drop
low-confidence detections and draw each one as an antialiased dot in
screen space through the same forward mapping as the markers: about
1 px at 1080p, growing slightly with the lens zoom, opacity from FRP on
a log scale between 1 and 500 MW. The colour and max blend under the
clouds stay as before.
Mercator at 16:9 only showed about +-71 degrees, cutting off northern
Greenland and the Arctic. The new variant world is plate carrée, north up,
centred on the equator and the home longitude, showing -90..90 degrees
stretched vertically to fill the screen. It replaces mercator in the CLI
--projection choices and the server's variant parameter, without an alias
for the old name. lens and plain are unchanged.
acheron gets BlueZ (adapter "gg23", UB500 dongle) as an A2DP sink only;
WirePlumber's bluez monitor targets received streams at the "hub" sink,
like roc-lan and AirPlay. A bt-agent (NoInputNoOutput) accepts pairing
without PIN. A Home Assistant button (MQTT discovery) opens a 120s
discoverable/pairable window; afterwards bonded devices are trusted so they
reconnect any time, and a binary_sensor shows the window state.
Replaces the farm call. eval evaluates the new ciToplevels output once
with hestia's matrix subaction (the same machines the farm built: all
nixosConfigurations), shared builds the derivations needed by more than
one uncached machine once and hands the hestia manifest version with
their outputs to the machine jobs, one job per uncached machine builds
its toplevel by drv path. The ci job stays the required check; shared
and the machine jobs are skipped when nothing is uncached.
Every job pushes the store paths it added, built or substituted from
hestia, to niks3 with its GitHub OIDC token, as the farm did, so
deploys find the toplevels in cache.lassul.us. The niks3 client is the
new packages.niks3 output.
Measured on the ci-hestia-shared experiment with a nixpkgs bump: 15 min
(eval 3.3, shared 3.3, 1-4 per machine) instead of 49 with the farm. No
more queueing of runs for the farm.
MCP server that controls the everyday Firefox through a WebExtension and
a native messaging host instead of Marionette/WebDriver, so
navigator.webdriver stays false and Cloudflare/reCAPTCHA stop flagging
normal browsing. No browser restart is needed to attach, tabs are
addressed by id on demand.
barnacle installs the server and links the native messaging host
manifest into /Library/Application Support/Mozilla/NativeMessagingHosts.
The extension is signed as unlisted on AMO via browser-mcp-sign.
atuin 18.23.0 fails this e2e test in unsandboxed darwin builds: the test
shell reads nix-darwin's /etc/bashrc, which resets PATH so the atuin
under test is not used. Fixed in nixpkgs by NixOS/nixpkgs#574509
(NOSYSBASHRC in preCheck); drop this package once our nixpkgs pin
includes it. Linux keeps the unmodified nixpkgs derivation.
ntfy.sh's free tier counts every UnifiedPush message against the
subscriber: 250 messages/day shared by all UnifiedPush apps on the phone.
Once used up it answers 429, phonetpm can no longer wake the phone and
every SSH login and deploy fails. Run our own ntfy on neoprism instead,
anonymous like ntfy.sh so app servers publish without accounts.
Rate limits: ntfy 2.29 defaults stay, except for an exemption.
- visitor-message-daily-limit is 0 (unlimited), only the request limiter
applies.
- visitor-subscriber-rate-limiting is off (ntfy.sh turns it on), so
publishes to up* topics are charged to the publisher, not to the phone.
The phone only spends requests on its subscription connections (30
allowed, the app uses one per server).
- The request limit is 60 burst, then one per 5s, per publisher IP.
Synapse on neoprism sends every Matrix push of the phone from one
address; encrypted and bridged rooms notify per message, so bridge
catch-up can exceed that. visitor-request-limit-exempt-hosts exempts
neoprism's own addresses. phonetpm daemons publish one wake per
connection and stay far below the limit.
- behind-proxy: ntfy takes the right-most X-Forwarded-For address, the
one nginx appends, so visitors are not all 127.0.0.1 and a client
cannot spoof its address.
The module enables attachments by default (attachment-cache-dir), set it
empty to keep them off. nginx settings follow ntfy's example config
(buffering off, 3m timeouts against the 45s keepalive, websockets).
No backup: the state dir (/var/lib/private/ntfy-sh) holds the message
cache (12h, disposable) and an auth db without users.
Since d9ed167 networkd installed the route to the container itself.
networkd adds it as soon as ve-riot gets carrier, before the container
start script runs `ip route add 10.233.1.2 dev ve-riot`, so the script
fails with "RTNETLINK answers: File exists" and container@riot ends up
in a restart loop. Before the reboot riot had never restarted since
d9ed167, so nobody noticed.
The script's route and address come from nixpkgs' nixos-containers
module and can't be turned off without patching nixpkgs (localAddress
also sets the container side), so the networkd route has to go. Now
ve-riot is Unmanaged=yes instead, and the start script owns its
address and route. Checked against systemd 261:
- An unmanaged link gets no configuration from networkd, so nothing is
in place before the script's `ip route add` and it succeeds on every
start.
- networkd only drops foreign routes for the link it is configuring
(link_drop_unmanaged_config, CONFIGURING state). When link_drop_routes
marks foreign routes it skips routes on other interfaces, "Otherwise,
routes assigned to unmanaged interfaces will be dropped". So a networkd
restart or reload no longer touches the script's route. That was the
bug d9ed167 fixed.
KeepConfiguration=static could not keep the script's route anyway: it
only protects foreign routes with protocol static, and `ip route add`
uses protocol boot. LinkLocalAddressing=no and KeepConfiguration=static
mean nothing on an unmanaged link, so they are gone, along with the
commented-out IPForward/IPMasquerade lines. Forwarding and NAT stay in
nftables and the sysctl. No other .network file on neoprism matches
ve-riot (99-ethernet-default-dhcp needs Kind=!*), but Unmanaged=yes
also keeps a later catch-all file from taking over ve-riot.
neoprism's NVMe disks report critical_warning 4 and a failing SMART
status only because percentage_used is above 100% (rated endurance
exceeded). Hetzner confirmed with the manufacturer that bit 0x04 is set
for that reason and that the drives are fine as long as available_spare
is above available_spare_threshold, which NvmeSpareLow watches. smartctl
derives smart_status from the same bit, so both alerts stayed red on
worn but healthy drives.
NvmeCriticalWarning now fires only on critical_warning bits other than
0x04 (bitmap_and with 251). SmartFailed skips devices that
smartctl_exporter reports with protocol="NVMe" and keeps covering
SATA/SCSI disks.
smartctl_exporter runs on every non-container machine, bound to
localhost and scraped by Vector into the host-tagged metrics path.
Alerts on a failed SMART status, NVMe critical warnings, new NVMe media
errors and NVMe spare below its threshold.
Memory PSI comes from /proc/pressure/memory via an exec source, as a
counter of stalled seconds, so the alert takes an exact 10m rate instead
of the kernel's 5m avg300 EWMA. Plus an alert on any OOM kill.
MemoryPressure threshold 10%: some task stalling on memory a tenth of
the time over 10 minutes is sustained reclaim/swap thrash, while short
spikes such as big builds average out.
autoUpgrade pins the upgrade to the revision the forward-only guard resolved and writes it to /var/lib/superconfig-revision after a successful upgrade, like deploy does. Vector reads that file every 30s and exports nixos_system_info{host,revision,revision_short}; the node dashboard shows host -> revision_short.
HTTP service rendering wallpapers on demand for a location, variant,
overlay and size. Renders stay fresh for 10 minutes; ones that were
picked up are re-rendered in the background when they go stale.
realwallpaper gains --cache-dir so locations share one data cache, and
its cache cleanup now tracks last use instead of fetch time.
Replaces pkgs.realwallpaper (xplanet, dead NASA NEO sources) with a
Python renderer: oblique cylindrical map centred on home with a zoom
lens (or plain/mercator), multi-resolution WMS imagery (Sentinel-2
cloudless, Blue Marble, Black Marble lights), live EUMETSAT clouds with
synthesised detail, VIIRS snow/fire/chlorophyll, sea ice and the sky.
The krebs.realwallpaper module, its nginx config and the mors
fetchWallpaper config are dropped; the tool runs standalone for now.
Branches, PRs and auto-merge now use the UPDATE_PKGS_TOKEN repo secret
instead of GITHUB_TOKEN, so the normal pull_request ci.yml run (with the
farm build) shows up on each PR and GitHub merges it with a merge commit
once the ruleset is met. The serial verify job (rebase, dispatch ci.yml,
wait, merge) and the final ci dispatch on master are gone: the auto-merge
counts as the token's, so the push to master triggers ci.yml itself.
UPDATE_PKGS_TOKEN: fine-grained PAT (or GitHub App token) for this repo
with Contents: read/write, Pull requests: read/write and Workflows:
read/write (force-pushing an update branch rebased over master commits that
touched .github/workflows is refused without it). The repo needs "Allow
auto-merge" enabled. The workflow fails early if the secret is missing.
Every NixOS machine runs Vector (host metrics, internal metrics, journal)
and writes to VictoriaMetrics/VictoriaLogs on both styx and coaxmetal,
each sink with a 512 MiB disk buffer. Both servers run vmalert (metrics and
LogsQL instances), a two-node Alertmanager cluster posting to Matrix via
matrix-alertmanager-receiver, Gatus (results scraped into VictoriaMetrics,
alerted on by vmalert) and Grafana with provisioned datasources and
dashboards. All ports are open on retiolum only.
Replaces telegraf on neoprism, including the borgbackup status file.