Commits
The I/O runs on a tokio runtime now:
- HTTP via reqwest (rustls on aws-lc, HTTP/2, bundled webpki roots as
before) instead of ureq. Instead of our own caching ureq::Resolver the
cache hosts are resolved once at startup and pinned with
resolve_to_addrs, which keeps getaddrinfo() (and NSS modules from the
store) out of requests once /nix/store is mounted. The retry loop
stays: reqwest-middleware/retry would not save code, as body read
errors are retried too.
- Decompression via async-compression (zstd, xz, bzip2, gzip, brotli,
concatenated members like Nix) instead of zstd/xz2/bzip2/flate2/
brotli-decompressor; the magic sniffing for narinfo/.ls stays.
- The closure walk and the .ls listings run with buffer_unordered
instead of par_map threads.
- FUSE workers stay threads; open() fetches via Handle::block_on, and
concurrent opens of a path wait on a tokio Mutex.
- The compressed size is counted with tokio-util's InspectReader instead
of our CountingReader. The NAR is still unpacked by our reader for now,
through SyncIoBridge.
Replace our nix32 encoder with harmonia-utils-base-encoding's base32 and
the hand-walked serde_json::Value of the caches' .ls listings with
harmonia-file-core's FileTree, which is the serde model of Nix's file
system object JSON. Both are pure crates (data-encoding/derive_more,
serde/serde_json/thiserror), pulled as a git dependency pinned to
harmonia 3edb37ae: only harmonia-utils-hash and harmonia-store-core exist
on crates.io, as 0.0.0-alpha.0 placeholders. A listing node without
"size" (regular) or "entries" (directory) is now an error instead of
being read as 0 / empty; Nix always writes both.
Kept ours:
- NAR unpacking: harmonia-file-nar is a tokio AsyncRead parser/restorer;
we unpack synchronously from the blocking HTTP/decompressor stream
while hashing, in a sync FUSE daemon.
- narinfo parsing, store path names and NarHash parsing:
harmonia-store-nar-info, -store-path and -utils-hash all pull in
aws-lc-rs (aws-lc-sys C build), blake3 and md5, nar-info also
ed25519-dalek and getrandom: +43 crates for ~15 lines. closure also
passes the narinfo text through verbatim, and harmonia's parser
rejects whole narinfos for fields we don't use (Sig, CA, FileHash).
- The in-memory Tree: matching FileTree(FileSystemObject::Regular(
Regular { .. })) in lazy.rs is clumsier than our flat enum.
Incident 2026-10-17: radio→styx UDP was blackholed for ~40 min after a
sendto() hit ENETUNREACH; tincd never recovered the UDP path on its own.
The fix is upstream as Mic92/tincr#140.
The fork is rebased onto upstream Mic92/tincr main: 6 of the 8 commits on
lassulus/fixes are upstream now, leaving the tunnel-address gate and the
ENETUNREACH fix on lassulus/fixes-rebased (bfcbbc69).
pkgs/tincr pins the same branch and is what every machine runs (the flake
input only provides barnacle's darwin module), so bump it and its update
script along with the input.
RUST_LOG=tinc=warn hid every routing/relay event, so path problems were
only visible over SSH. Enable Info for tincd::graph (became
reachable/unreachable via X), tincd::conn (meta connections up/down) and
tincd::net (key exchange, UDP idle revalidation/timeouts, PMTU
fixed/increase/decrease). Their Debug/Trace stays off because tincd::net
and tincd::proto log per packet there; tincd (autoconnect), tincd::auth
and tincd::proto Info stay off as redundant.
Expected volume is roughly 100-250 lines/h on a hub like neoprism and
30-60/h on leaves, plus autoconnect dial attempts to dead hosts in conn.
iets (numtide/iets, a Nix evaluator in Rust) gives the same drvPath,
isCached and cacheStatus for every machine as nix-eval-jobs 2.35.4, in
24-35s instead of 108-144s (4 workers, coaxmetal). It is a flake input
so CI builds it once and pushes it to cache.lassul.us with niks3 and
the lazy store.
Reading the lock file from the fetched src is import from derivation:
every evaluation of the machines that have it (aergia, green, ignavia,
mors) fetches the source first, and evaluators without IFD (iets) fail
on them.
Account linking is off: byoidc never marks emails verified and anyone can
run a provider claiming any address. New users are created automatically.
idps.<name>.type = "oidc" with an issuer, a literal clientId as
alternative to clientIdFile, a display name independent of the attribute
name and per-IdP providerOptions (default unchanged: linking, creation and
auto update allowed). Generic OIDC IdPs use PKCE and ID token mapping.
Zitadel 2.71 does not fetch the issuer's discovery document when the IdP
is created, only at login, so the reconciler needs no ordering on the
provider.
byoidc lets people sign in with their own OpenID provider; Zitadel will
offer it as one generic OIDC IdP. The downstream client "zitadel" gets its
secret from a prompt-less clan var, which root-owned is fine for both
byoidc (LoadCredential) and zitadel-reconcile (root).
Its state (cached upstream clients, signing key) is not backed up: subjects
are derived from issuer and upstream sub, a new signing key is picked up
through JWKS, and upstream clients are registered again on demand.
id.lassul.us is Zitadel now; its endpoints are /oauth/v2/authorize,
/oauth/v2/token and /oidc/v1/userinfo (from its discovery document). The
pad project is public: every registered user may log in. zitadel-reconcile
creates the HedgeDoc app and writes its client id/secret into the env file
HedgeDoc reads, so there is no prompt var for them; hedgedoc starts after
it. HedgeDoc posts the client secret in the token request body, hence the
POST auth method.
lass.zitadel describes Zitadel's login policy, its external IdPs and one
project per service with OIDC apps and an access mode:
- public: no "check authorization on authentication", every registered
user may log in.
- declarative: the check is on and the project's user grants are exactly
the listed users (login name or verified email address). Removing a
user from the list revokes the grant; grants given in the console are
removed as well.
- managed: the check is on, the grants are left to the console.
zitadel-reconcile, a oneshot after zitadel, applies it at boot and on
every deploy that changes it. It authenticates with the PAT of a service
user (IAM_OWNER) that zitadel's setup creates with the first instance
(FirstInstance.Org.Machine, PatPath). Setup writes that file world
readable, so /var/lib/zitadel becomes a 0700 state directory. An empty Pat
section creates no token, hence the explicit expiration date (Zitadel's
own default).
A small Python script against the admin/management v1 REST APIs rather
than OpenTofu with the zitadel provider:
- "declarative" has to own all grants of a project, including ones made
in the console; terraform only removes what is in its state.
- Users are listed before they register. A failed user lookup would fail
the whole plan; the script skips such users until the next run.
- No state file that has to match Zitadel: objects are found by name, so
e.g. an app deleted in the console is created again.
- The provider talks gRPC to the external domain, through nginx and TLS;
the script talks plain HTTP to 127.0.0.1 with the Host header.
- Projects and apps removed from the config stay in Zitadel instead of
being destroyed together with their grants.
Zitadel shows a client secret only when it creates one. The script keeps a
root-only copy per app in /var/lib/zitadel-apps and renders the app's env
file from it (path, owner and variable names configurable); without a copy
for the current client id it generates a new secret. Services on other
hosts find their secret there too, for now.
The login policy moves from DefaultInstance, which only applies when the
instance is created, to lass.zitadel.loginPolicy. GitHub and Google logins
come from persisted prompt vars; an empty prompt leaves the IdP out, so
the first deploy works before the OAuth apps exist. Their callback is
/ui/login/login/externalidp/callback (login UI v1, the default in 2.71).
Users may create accounts with them and link them to existing ones.
The script waits for /admin/v1/healthz rather than /debug/ready: the HTTP
gateway's connection to the gRPC API comes up later.
Zitadel 2.71 from nixpkgs as a federated IdP with open self-registration:
registration, external IdPs and passkeys allowed in the default login
policy, first instance "lassul.us" with admin user lass (lass@lassul.us).
The masterkey and the admin's initial password come from the zitadel clan
vars generator, the password via an extra steps file so it never hits the
store.
Postgres via peer auth over the local socket. ensureDatabases/ensureUsers
create the database and owner role, so the zitadel role gets no
CREATEROLE/CREATEDB (on PG15 CREATEROLE is close to superuser). That needs
`init zitadel` + start-from-setup instead of the module's start-from-init,
whose init always tries CREATE USER/CREATE DATABASE and fails on the
permission check before the "already exists" check.
nginx terminates TLS and talks h2c to Zitadel via grpc_pass, as Zitadel's
reverse-proxy docs require for its gRPC APIs. Machine identification by
hostname, since the default private-IP/GCP-webhook probes don't fit
neoprism.
The database is pg_dumped by the borg preHook; gatus checks
/debug/healthz. Zitadel 2.71 does not work with PostgreSQL 18 (migration
34_add_cache_schema creates unlogged partitioned tables), neoprism runs 15.
For when.lassul.us (timeful) to show lass's availability without event
details, at https://cal.lassul.us/busy/<token>.ics.
With --busy-token-file, GET /busy/<token>.ics returns the busy time book
itself uses for free slots (CalDAV.busy plus held requests), so the feed
and the booking page agree on what blocks time: recurrences expanded,
all-day events from local midnight, TRANSPARENT and CANCELLED events
left out, TENTATIVE events (own ones and pending booking requests) kept
as busy, since book holds those slots too.
Intervals are merged so overlapping events cannot be counted, and each
VEVENT carries only DTSTART/DTEND in UTC, DTSTAMP (required by RFC
5545), SUMMARY "Busy", TRANSP OPAQUE and a UID hashed from its times.
The window runs from 7 days back to 90 days ahead: book's own 28 day
horizon is too short for polls planned weeks ahead, and the past week
keeps polls that are still open showing sensible data.
The feed is generated per request, so it is as fresh as the calendar;
an unreadable calendar gives 503, never an empty feed. Any other path
under /busy/ gives 404; the token is compared in constant time.
configs/book.nix: the token comes from the prompt-less vars generator
busy-feed and reaches book as a systemd credential, so it never lands in
the Nix store; nginx proxies all of cal.lassul.us/busy/ to book.
book.lassul.us returns 404 for /busy/ so the feed has a single URL.
In the benchmark run (PR 68) the neoprism job spent 29 s between
fetching lazy-store and mounting: the dry-run for the paths to
substitute (13 s in the shared job for the same machine) and their
narinfo closure (the rest, 2,878 paths), both after the 21 s hestia
prefetch of the drvs, which the dry-run needs.
The eval job has the drvs and dry-runs the machines anyway for the
shared derivations; it now keeps each machine's "will be fetched" list
and uploads it with the shared ones. The machine job collects the
closure of that list in the background while hestia prefetches the
drvs, then mounts. Computing the closure in the eval job too would only
move its time onto the eval job, before every machine job starts.
For a single machine the eval job now dry-runs it (about 13 s), against
about 29 s less in the machine job; with more machines the dry-runs ran
already, and in parallel.
The shared job dry-ran every machine to find the derivations needed by
more than one, after a runner start and a prefetch of all drv closures
from hestia. In the benchmark run (PR 68, only neoprism uncached) it
built nothing and still cost the machine job 61 s: 5 s queueing, 6 s
setup, 21 s prefetch (mostly the 47 MiB nixpkgs source), 13 s dry-run,
then 6 s queueing of the machine job.
The eval job has the drvs already, so it dry-runs the machines itself,
in parallel, and only for two or more (one machine shares nothing). It
hands the list to the shared job as an artifact; with an empty list the
shared job is skipped, and the machine jobs wait for the manifest
version of the eval job instead of the shared job's.
`nix run nixpkgs#nix-eval-jobs` resolves nixpkgs through the global
registry (nixpkgs-unstable): a second nixpkgs, fetched as a GitHub
tarball and unpacked into the git cache on every eval job, next to the
locked one the flake evaluation substitutes from cache.nixos.org
anyway. A cold nixpkgs tarball fetch took 18.6 s locally; on the
runner it hides inside the 99 s matrix step of the benchmark run
(PR 68), which prints no progress. With --inputs-from . the eval job
builds nix-eval-jobs from the locked nixpkgs (substituted from
cache.nixos.org) in the tools step and hands the binary to the matrix
step.
The worker count stays at 4: locally nix-eval-jobs over ciToplevels
took 101 s with 4 workers (1.8 cores busy), 72-83 s with 8 and 137 s
with 6 x 2 GiB, too noisy on a shared 16-core host to say anything
about the 4-vCPU runner.
audio-matrix (./matrix.py) watches PipeWire (pw-dump --monitor) and
publishes per source and room an MQTT discovery switch
switch.audio_route_<source>_<room> (attributes source, speaker, kind),
grouped as one HA device per source. Sources are AirPlay (shairport's
"Shairport Sync" stream), LAN (roc-lan) and every connected Bluetooth
phone (bluez_input.*, keyed by MAC, named by its alias); a phone's switches
are removed when it disconnects. The daemon links the source node directly
to roc-<room> with pw-link and re-applies the links whenever nodes come
back (phone reconnects, PipeWire restarts).
The routes are kept per source key in /var/lib/audio-matrix/routes.json, so
a reconnecting phone gets its previous routing. A source never seen before
plays on all rooms: a freshly paired phone should just make sound.
The hub null sink, the "Audio room" select, its buttons, the audio-room
service and command are gone. WirePlumber must not link the sources itself
any more: node.autoconnect = false for bluez_input.* (monitor.bluez.rules
run after name-node.lua sets it true) and roc-lan (source.props), and
PIPEWIRE_AUTOCONNECT=false for shairport-sync, whose pipewire backend
always connects with PW_STREAM_FLAG_AUTOCONNECT. Groups without the null
sink are driven by the phone's node or PipeWire's Dummy-Driver.
The old discovery configs stay retained on mosquitto; clear them once on
acheron:
for t in homeassistant/select/audio_room/config \
homeassistant/button/audio_room_{shodan,icarus,both,off}/config \
gg23/audio-room/set gg23/audio-room/state; do
mosquitto_pub -h localhost -u gg23 -P gg23-mqtt -r -n -t "$t"
done
A Pixel 9a with HD audio negotiated a high-quality codec, and WirePlumber
failed to create the bluez_input node ("mod.adapter: can't get format:
Invalid argument"), so no audio reached the hub and the phone retried
every second. With SBC it works. AAC decoding is available: the
nixpkgs pipewire bluez5 aac plugin links fdk-aac. sbc_xq shares the
sbc endpoint and has a decoder.
llm-agents now ships orca 1.4.227, whose release exists again, so the
1.4.224 pin (for the deleted 1.4.225/1.4.226 releases) and its assert
are no longer needed.
yellow mounts the same ZFS dataset (tank/download) as neoprism's
/var/download, which is covered by DownloadSpace*, so DiskAlmostFull
fired for it. Match the exemption on the device of neoprism's series
instead of host and mountpoint.
sync-containers3 containers (radio, yellow) get the host's
/nix/var/nix/db bind-mounted read-only, so nixos-upgrade failed every
night with "opening lock file /nix/var/nix/db/big-lock: Read-only file
system". Talk to the bind-mounted daemon socket instead, like the
autoswitch service does. Only for boot.isContainer, so other machines
are unaffected by that part.
The guard reported "cannot reach" for any nix flake metadata failure,
including this one; say what actually happened instead. nix's own
error is already on stderr above it.
Load the hermes-voice plugin into the gateway (matrix_dm_send, end_call,
/assist), and enable the gateway's webhook platform on 127.0.0.1:8644
with a deliver_only route to the Matrix home room, through which
hermes-voice asks approvals that no call can hear. The route is signed
with WEBHOOK_SECRET, which hermes-env fills with the API server key that
hermes-voice already holds: no new secret.
gpt-live-1 with client delegation has no client-side function tools: the
only model-to-page signal is session.delegation.created, which carries no
name or arguments. So the voice model does not hang up itself: goodbye,
"that's all", "hang up" and "I'm leaving" are delegated like any request,
and Hermes decides. BACKEND_NOTE tells it to call end_call once requested
work has been started or handed off, never while it still needs an answer
such as an approval.
end_call lives in the gateway plugin. Plugin handlers get the calling
session_id (model_tools.py passes it to registry.dispatch), and the
session context names the platform. The tool succeeds only on the API
server in a session whose lineage starts at a voice-* id: context
compression continues a session under a new id with the old one as
parent_session_id, so the prefix alone would not survive a long call.
Elsewhere it returns an error and changes nothing.
hermes-voice sees the tool in the run's events (tool.completed of
end_call without error) and passes an end_call item to the page instead
of a tool item. The page hands Hermes's reply to the voice model as
usual and closes the session once the voice has spoken it and been quiet
for 800 ms, at the latest after 30 s; close() now runs only once. A run
no page listens to (hang-up flush, page gone) only logs it.
Hermes registers no agent-callable send_message tool on any platform
(tools/send_message_tool.py: "intentionally NOT registered as an
agent-callable model tool"; toolsets.py: "deliberately no agent-callable
send_message tool"), so there is no config switch, platform toolsets
included, that lets voice sessions on the API server write to Matrix. The
plugin adds matrix_dm_send, limited to the Matrix home room (the owner's
DM) and sent through the gateway's live adapter by Hermes's own send
helper, which also mirrors the message into the DM's session. Reading the
DM needs nothing new: session_search (chat_history_lookup) already reads
the Matrix session's transcript from any session.
/assist approve|deny [code] answers an approval hermes-voice asked in the
DM. A Matrix /approve cannot do it: it only resolves approvals of the
chat's own session, and API runs are keyed by run id. /voice is taken by
Hermes itself.
Hermes does not stop a run when the /v1/runs/{id}/events subscriber goes
away (the SSE handler only detaches it), so runs already survived the
page. But nothing listened any more: an approval request then waited for
nobody and was denied by Hermes's timeout. hermes-voice now follows every
run it starts in its own thread, page or no page; the page replays the
run's items and gets blank keepalive lines so a vanished page is noticed.
On hang-up (button, idle, page close) the page posts /hangup with
keepalive: runs of the conversation count as unheard from then on, and
what the user said since the last delegation becomes a run of its own
with a note that nobody listens. Lines that the model's delegation
already took are not sent again, and a delegation still in its settle
delay leaves them to the hang-up. Only a correction within the same call
(replace) stops the previous run; a new call no longer stops work an
earlier call started.
An approval no call can answer is posted to the owner's Matrix DM (the
gateway's Matrix home room) through the gateway's webhook platform
(--matrix-webhook, signed with the API server key) with a short code;
/approve takes that code, from the page or from the Matrix side with the
API server key.
Approvals also failed outright before: under the gateway's session
authority the approval request is named prompt_id and the answer must be
exactly {request_id, execution_generation, choice}, while hermes-voice
sent only {choice} (invalid_params).
The persona and the server's texts name no particular user: they speak
of "the user" and of the owner's Matrix DM.
voice.lassul.us goes away without a redirect, like other retired names
in the zone. There is no gatus check for it (coaxmetal is not alwaysOn).
With the lazy /nix/store in the machine jobs, hestia's drain read every
cache.lassul.us-only input path to mirror it into the GitHub Actions
cache, downloading 375-760 MiB per job through the lazy store and
spending 1-4.5 min uploading to the Actions cache at 2-11 MiB/s (PR 65
run). Those paths are already in cache.lassul.us; niks3 uploads
everything newly built.
upstream-cache-key-names replaces hestia's default list, so it repeats
cache.nixos.org-1 next to cache.lassul.us-1. In the eval job,
filter-drv-closures applies the same keys to the drv closures; members
signed by cache.lassul.us are substituted from there by the later jobs.
The day side now shows what the geostationary satellites see, like
zoom.earth, instead of the base map under white clouds synthesised from
infrared:
- Europe/Africa/Atlantic: MTG (Meteosat-12) rgb_truecolour from EUMETSAT.
rgb_geocolour looks the same by day but renders ~4x slower on the
server and timed out (502) for full 2048 px tiles.
- Americas/Pacific: GOES-East and GOES-West GeoColor from GIBS.
- Asia/Australia: GIBS has no Himawari colour product. Himawari band 3
(red, 10 min) is turned into white cloud over the base map, with the
cloud opacity being the reflectance beyond the ground's own albedo.
The daily VIIRS true colour mosaics were the alternative, but their
clouds are up to a day old, have swath seams of different times and
would not match the IR clouds next to them.
Each source is weighted by how well its satellite sees the point: full
trust up to 62° view zenith, none from 78° (the limb is smeared), and
the more nadir satellite wins in overlaps with cos^8 of the zenith
angle, so seams are soft. Trust also needs the sun not to be in front
of the satellite: looking towards a low sun both MTG (mornings over the
Indian Ocean) and Himawari (evenings) are washed out to a pale band
with straight edges, while a long path with the sun behind the
satellite is fine (Berlin at winter noon). A plain path-length limit
would have dropped the latter. Near the terminator (cosz < 0.2)
GeoColor fades into its purple night blend, so the live imagery fades
out there and the IR clouds over the base map take over as before.
Where it is shown, the synthetic clouds and so their relief shading and
shadows are scaled down, so the real image is not shaded twice.
All live layers (including the IR clouds) now request an explicit TIME,
the render time floored to 10 min: without TIME GIBS served tiles of
one layer from different frames (sun glint cut at a tile edge), and it
makes --time render the imagery of that time, not only the sun. The
TIME is capped at the newest frame the server lists (GIBS
DescribeDomains, EUMETSAT per-layer capabilities), since GIBS answers
an unpublished time with an empty image and EUMETSAT with an error;
MTG true colour stops producing frames while its disk is dark.
GIBS also now and then answers with an empty 1 kB image for a frame it
has (the same request succeeds when repeated), which left a quarter of
the Himawari disk missing, so empty tiles are asked for up to 3 times.
A machine job substituted the whole input closure of the derivations it
builds, e.g. 3.4 GiB for neoprism, of which the builds open a few dozen
MiB. .github/lazy-store.sh now runs before the build: it takes what
`nix build --dry-run` would fetch, collects its closure from the
narinfos of cache.nixos.org and cache.lassul.us (tools.lazy-store),
mounts lazy-store on /nix/store, registers the paths with
`nix-store --load-db` and copies their signatures. The build itself,
ci-build.sh's niks3 push and hestia stay as they were: outputs are
written to the real store dir below the mount, and ci-build.sh's
before/after diff of the store starts after the registration.
If anything fails before the paths are registered, it logs an error,
undoes the mount and exits 0, so the build substitutes eagerly. The FUSE
log and the fetched NARs are printed by an always-run step after the
build.
The eval job builds lazy-store next to niks3 and pushes it, the machine
jobs substitute it. The eval and shared jobs keep the plain store: eval
builds nothing but niks3 and lazy-store, and shared builds package
derivations, which read most of their inputs anyway.
FUSE file system for CI's machine builds, ported from the experiment on
ci-lazy-fuse (.github/exp/lazy-store). It mounts on /nix/store the real
store dir (all writes go there) merged with store paths of
cache.nixos.org and cache.lassul.us, whose metadata comes from their
.ls listings and whose NAR is fetched, checked and unpacked on the first
open() of a file in them; reads go through kernel FUSE passthrough. In
the experiment, rebuilding neoprism's system-units, etc, system-path and
toplevel downloaded 62 MiB instead of the 3,455 MiB of an eager
substitution.
Changes from the experiment:
- closure takes the store paths to make valid on stdin (what
`nix build --dry-run` would fetch) instead of walking derivations, and
leaves out paths whose closure is not entirely in the caches, so that
nix never sees a valid path with a missing reference; nix substitutes
those as usual.
- closure also writes the fetched narinfos as a file:// binary cache:
`nix-store --load-db` registers no signatures, and niks3's
--ignore-upstream-key and hestia's upstream-cache-filter go by the
signatures in the local database, so without them both would upload
(and thereby download) every cache.nixos.org path in an output's
closure. `nix store copy-sigs` adds them from there.
- mount fails if a path has no listing instead of leaving it out, as
every path gets registered.
Static, so that once it serves /nix/store it loads nothing from there.
A flake package, so CI builds it once and pushes it to cache.lassul.us,
and the jobs substitute it. passthru.tests.fetch covers fetch, closure
and the registration with signatures against a file:// cache.
The GIBS VIIRS_NOAA20_Thermal_Anomalies_375m_All raster shows each
detection as a coarse blob, far bigger than its 375 m footprint at
wallpaper scale, so the fire layer was loud and distracting.
Fetch FIRMS' keyless global 24 h VIIRS CSVs of NOAA-20, NOAA-21 and
Suomi NPP instead (cached, refreshed hourly like the old layer), drop
low-confidence detections and draw each one as an antialiased dot in
screen space through the same forward mapping as the markers: about
1 px at 1080p, growing slightly with the lens zoom, opacity from FRP on
a log scale between 1 and 500 MW. The colour and max blend under the
clouds stay as before.
Mercator at 16:9 only showed about +-71 degrees, cutting off northern
Greenland and the Arctic. The new variant world is plate carrée, north up,
centred on the equator and the home longitude, showing -90..90 degrees
stretched vertically to fill the screen. It replaces mercator in the CLI
--projection choices and the server's variant parameter, without an alias
for the old name. lens and plain are unchanged.
bluetoothd's hostname plugin overrode General.Name (and Class) with the
hostname and chassis, so disable it. bluetooth-pairing set adapter
properties before AutoEnable powered hci0, failed with Not Powered and
failed the switch; wait for Powered first.
acheron gets BlueZ (adapter "gg23", UB500 dongle) as an A2DP sink only;
WirePlumber's bluez monitor targets received streams at the "hub" sink,
like roc-lan and AirPlay. A bt-agent (NoInputNoOutput) accepts pairing
without PIN. A Home Assistant button (MQTT discovery) opens a 120s
discoverable/pairable window; afterwards bonded devices are trusted so they
reconnect any time, and a binary_sensor shows the window state.
ci.yml builds the machines with hestia now. Drops farm.yml, tools/ci-farm, neoprism's nix-grpc-daemon builder, barnacle's farm client, the nix-farm vars generators and the nix-grpc-store input. cache.lassul.us/niks3 moves to configs/nix-cache.nix.
Daily garbage collection of the hestia cache on the default branch, after
hestia's own gc.yml, so ci.yml's cache stays within the 10 GB GHA
cache quota.
Replaces the farm call. eval evaluates the new ciToplevels output once
with hestia's matrix subaction (the same machines the farm built: all
nixosConfigurations), shared builds the derivations needed by more than
one uncached machine once and hands the hestia manifest version with
their outputs to the machine jobs, one job per uncached machine builds
its toplevel by drv path. The ci job stays the required check; shared
and the machine jobs are skipped when nothing is uncached.
Every job pushes the store paths it added, built or substituted from
hestia, to niks3 with its GitHub OIDC token, as the farm did, so
deploys find the toplevels in cache.lassul.us. The niks3 client is the
new packages.niks3 output.
Measured on the ci-hestia-shared experiment with a nixpkgs bump: 15 min
(eval 3.3, shared 3.3, 1-4 per machine) instead of 49 with the farm. No
more queueing of runs for the farm.
levenshtein-merge only merged the managed CLAUDE.md for the wrapper.
coaxmetal's hermes keeps the plain llm.claude-code CLI.
stockholm's kartei is a git submodule since bc08283, and github: flake
refs never fetch submodules, so evaluating #hotdog failed with
kartei/default.nix missing.
MCP server that controls the everyday Firefox through a WebExtension and
a native messaging host instead of Marionette/WebDriver, so
navigator.webdriver stays false and Cloudflare/reCAPTCHA stop flagging
normal browsing. No browser restart is needed to attach, tabs are
addressed by id on demand.
barnacle installs the server and links the native messaging host
manifest into /Library/Application Support/Mozilla/NativeMessagingHosts.
The extension is signed as unlisted on AMO via browser-mcp-sign.
atuin 18.23.0 fails this e2e test in unsandboxed darwin builds: the test
shell reads nix-darwin's /etc/bashrc, which resets PATH so the atuin
under test is not used. Fixed in nixpkgs by NixOS/nixpkgs#574509
(NOSYSBASHRC in preCheck); drop this package once our nixpkgs pin
includes it. Linux keeps the unmodified nixpkgs derivation.
ntfy.sh's free tier counts every UnifiedPush message against the
subscriber: 250 messages/day shared by all UnifiedPush apps on the phone.
Once used up it answers 429, phonetpm can no longer wake the phone and
every SSH login and deploy fails. Run our own ntfy on neoprism instead,
anonymous like ntfy.sh so app servers publish without accounts.
Rate limits: ntfy 2.29 defaults stay, except for an exemption.
- visitor-message-daily-limit is 0 (unlimited), only the request limiter
applies.
- visitor-subscriber-rate-limiting is off (ntfy.sh turns it on), so
publishes to up* topics are charged to the publisher, not to the phone.
The phone only spends requests on its subscription connections (30
allowed, the app uses one per server).
- The request limit is 60 burst, then one per 5s, per publisher IP.
Synapse on neoprism sends every Matrix push of the phone from one
address; encrypted and bridged rooms notify per message, so bridge
catch-up can exceed that. visitor-request-limit-exempt-hosts exempts
neoprism's own addresses. phonetpm daemons publish one wake per
connection and stay far below the limit.
- behind-proxy: ntfy takes the right-most X-Forwarded-For address, the
one nginx appends, so visitors are not all 127.0.0.1 and a client
cannot spoof its address.
The module enables attachments by default (attachment-cache-dir), set it
empty to keep them off. nginx settings follow ntfy's example config
(buffering off, 3m timeouts against the 45s keepalive, websockets).
No backup: the state dir (/var/lib/private/ntfy-sh) holds the message
cache (12h, disposable) and an auth db without users.
SystemdUnitFailed matched any line containing "Failed with result", so a
VictoriaLogs warning echoing a rejected query fired it for
victorialogs.service. Anchor a regexp on _msg to systemd's
"<unit>: Failed with result '<result>'." form.
Since d9ed167 networkd installed the route to the container itself.
networkd adds it as soon as ve-riot gets carrier, before the container
start script runs `ip route add 10.233.1.2 dev ve-riot`, so the script
fails with "RTNETLINK answers: File exists" and container@riot ends up
in a restart loop. Before the reboot riot had never restarted since
d9ed167, so nobody noticed.
The script's route and address come from nixpkgs' nixos-containers
module and can't be turned off without patching nixpkgs (localAddress
also sets the container side), so the networkd route has to go. Now
ve-riot is Unmanaged=yes instead, and the start script owns its
address and route. Checked against systemd 261:
- An unmanaged link gets no configuration from networkd, so nothing is
in place before the script's `ip route add` and it succeeds on every
start.
- networkd only drops foreign routes for the link it is configuring
(link_drop_unmanaged_config, CONFIGURING state). When link_drop_routes
marks foreign routes it skips routes on other interfaces, "Otherwise,
routes assigned to unmanaged interfaces will be dropped". So a networkd
restart or reload no longer touches the script's route. That was the
bug d9ed167 fixed.
KeepConfiguration=static could not keep the script's route anyway: it
only protects foreign routes with protocol static, and `ip route add`
uses protocol boot. LinkLocalAddressing=no and KeepConfiguration=static
mean nothing on an unmanaged link, so they are gone, along with the
commented-out IPForward/IPMasquerade lines. Forwarding and NAT stay in
nftables and the sysctl. No other .network file on neoprism matches
ve-riot (99-ethernet-default-dhcp needs Kind=!*), but Unmanaged=yes
also keeps a later catch-all file from taking over ve-riot.
neoprism's NVMe disks report critical_warning 4 and a failing SMART
status only because percentage_used is above 100% (rated endurance
exceeded). Hetzner confirmed with the manufacturer that bit 0x04 is set
for that reason and that the drives are fine as long as available_spare
is above available_spare_threshold, which NvmeSpareLow watches. smartctl
derives smart_status from the same bit, so both alerts stayed red on
worn but healthy drives.
NvmeCriticalWarning now fires only on critical_warning bits other than
0x04 (bitmap_and with 251). SmartFailed skips devices that
smartctl_exporter reports with protocol="NVMe" and keeps covering
SATA/SCSI disks.
smartctl_exporter runs on every non-container machine, bound to
localhost and scraped by Vector into the host-tagged metrics path.
Alerts on a failed SMART status, NVMe critical warnings, new NVMe media
errors and NVMe spare below its threshold.
Memory PSI comes from /proc/pressure/memory via an exec source, as a
counter of stalled seconds, so the alert takes an exact 10m rate instead
of the kernel's 5m avg300 EWMA. Plus an alert on any OOM kill.
MemoryPressure threshold 10%: some task stalling on memory a tenth of
the time over 10 minutes is sustained reclaim/swap thrash, while short
spikes such as big builds average out.
The I/O runs on a tokio runtime now:
- HTTP via reqwest (rustls on aws-lc, HTTP/2, bundled webpki roots as
before) instead of ureq. Instead of our own caching ureq::Resolver the
cache hosts are resolved once at startup and pinned with
resolve_to_addrs, which keeps getaddrinfo() (and NSS modules from the
store) out of requests once /nix/store is mounted. The retry loop
stays: reqwest-middleware/retry would not save code, as body read
errors are retried too.
- Decompression via async-compression (zstd, xz, bzip2, gzip, brotli,
concatenated members like Nix) instead of zstd/xz2/bzip2/flate2/
brotli-decompressor; the magic sniffing for narinfo/.ls stays.
- The closure walk and the .ls listings run with buffer_unordered
instead of par_map threads.
- FUSE workers stay threads; open() fetches via Handle::block_on, and
concurrent opens of a path wait on a tokio Mutex.
- The compressed size is counted with tokio-util's InspectReader instead
of our CountingReader. The NAR is still unpacked by our reader for now,
through SyncIoBridge.
Replace our nix32 encoder with harmonia-utils-base-encoding's base32 and
the hand-walked serde_json::Value of the caches' .ls listings with
harmonia-file-core's FileTree, which is the serde model of Nix's file
system object JSON. Both are pure crates (data-encoding/derive_more,
serde/serde_json/thiserror), pulled as a git dependency pinned to
harmonia 3edb37ae: only harmonia-utils-hash and harmonia-store-core exist
on crates.io, as 0.0.0-alpha.0 placeholders. A listing node without
"size" (regular) or "entries" (directory) is now an error instead of
being read as 0 / empty; Nix always writes both.
Kept ours:
- NAR unpacking: harmonia-file-nar is a tokio AsyncRead parser/restorer;
we unpack synchronously from the blocking HTTP/decompressor stream
while hashing, in a sync FUSE daemon.
- narinfo parsing, store path names and NarHash parsing:
harmonia-store-nar-info, -store-path and -utils-hash all pull in
aws-lc-rs (aws-lc-sys C build), blake3 and md5, nar-info also
ed25519-dalek and getrandom: +43 crates for ~15 lines. closure also
passes the narinfo text through verbatim, and harmonia's parser
rejects whole narinfos for fields we don't use (Sig, CA, FileHash).
- The in-memory Tree: matching FileTree(FileSystemObject::Regular(
Regular { .. })) in lazy.rs is clumsier than our flat enum.
Incident 2026-10-17: radio→styx UDP was blackholed for ~40 min after a
sendto() hit ENETUNREACH; tincd never recovered the UDP path on its own.
The fix is upstream as Mic92/tincr#140.
The fork is rebased onto upstream Mic92/tincr main: 6 of the 8 commits on
lassulus/fixes are upstream now, leaving the tunnel-address gate and the
ENETUNREACH fix on lassulus/fixes-rebased (bfcbbc69).
pkgs/tincr pins the same branch and is what every machine runs (the flake
input only provides barnacle's darwin module), so bump it and its update
script along with the input.
RUST_LOG=tinc=warn hid every routing/relay event, so path problems were
only visible over SSH. Enable Info for tincd::graph (became
reachable/unreachable via X), tincd::conn (meta connections up/down) and
tincd::net (key exchange, UDP idle revalidation/timeouts, PMTU
fixed/increase/decrease). Their Debug/Trace stays off because tincd::net
and tincd::proto log per packet there; tincd (autoconnect), tincd::auth
and tincd::proto Info stay off as redundant.
Expected volume is roughly 100-250 lines/h on a hub like neoprism and
30-60/h on leaves, plus autoconnect dial attempts to dead hosts in conn.
idps.<name>.type = "oidc" with an issuer, a literal clientId as
alternative to clientIdFile, a display name independent of the attribute
name and per-IdP providerOptions (default unchanged: linking, creation and
auto update allowed). Generic OIDC IdPs use PKCE and ID token mapping.
Zitadel 2.71 does not fetch the issuer's discovery document when the IdP
is created, only at login, so the reconciler needs no ordering on the
provider.
byoidc lets people sign in with their own OpenID provider; Zitadel will
offer it as one generic OIDC IdP. The downstream client "zitadel" gets its
secret from a prompt-less clan var, which root-owned is fine for both
byoidc (LoadCredential) and zitadel-reconcile (root).
Its state (cached upstream clients, signing key) is not backed up: subjects
are derived from issuer and upstream sub, a new signing key is picked up
through JWKS, and upstream clients are registered again on demand.
id.lassul.us is Zitadel now; its endpoints are /oauth/v2/authorize,
/oauth/v2/token and /oidc/v1/userinfo (from its discovery document). The
pad project is public: every registered user may log in. zitadel-reconcile
creates the HedgeDoc app and writes its client id/secret into the env file
HedgeDoc reads, so there is no prompt var for them; hedgedoc starts after
it. HedgeDoc posts the client secret in the token request body, hence the
POST auth method.
lass.zitadel describes Zitadel's login policy, its external IdPs and one
project per service with OIDC apps and an access mode:
- public: no "check authorization on authentication", every registered
user may log in.
- declarative: the check is on and the project's user grants are exactly
the listed users (login name or verified email address). Removing a
user from the list revokes the grant; grants given in the console are
removed as well.
- managed: the check is on, the grants are left to the console.
zitadel-reconcile, a oneshot after zitadel, applies it at boot and on
every deploy that changes it. It authenticates with the PAT of a service
user (IAM_OWNER) that zitadel's setup creates with the first instance
(FirstInstance.Org.Machine, PatPath). Setup writes that file world
readable, so /var/lib/zitadel becomes a 0700 state directory. An empty Pat
section creates no token, hence the explicit expiration date (Zitadel's
own default).
A small Python script against the admin/management v1 REST APIs rather
than OpenTofu with the zitadel provider:
- "declarative" has to own all grants of a project, including ones made
in the console; terraform only removes what is in its state.
- Users are listed before they register. A failed user lookup would fail
the whole plan; the script skips such users until the next run.
- No state file that has to match Zitadel: objects are found by name, so
e.g. an app deleted in the console is created again.
- The provider talks gRPC to the external domain, through nginx and TLS;
the script talks plain HTTP to 127.0.0.1 with the Host header.
- Projects and apps removed from the config stay in Zitadel instead of
being destroyed together with their grants.
Zitadel shows a client secret only when it creates one. The script keeps a
root-only copy per app in /var/lib/zitadel-apps and renders the app's env
file from it (path, owner and variable names configurable); without a copy
for the current client id it generates a new secret. Services on other
hosts find their secret there too, for now.
The login policy moves from DefaultInstance, which only applies when the
instance is created, to lass.zitadel.loginPolicy. GitHub and Google logins
come from persisted prompt vars; an empty prompt leaves the IdP out, so
the first deploy works before the OAuth apps exist. Their callback is
/ui/login/login/externalidp/callback (login UI v1, the default in 2.71).
Users may create accounts with them and link them to existing ones.
The script waits for /admin/v1/healthz rather than /debug/ready: the HTTP
gateway's connection to the gRPC API comes up later.
Zitadel 2.71 from nixpkgs as a federated IdP with open self-registration:
registration, external IdPs and passkeys allowed in the default login
policy, first instance "lassul.us" with admin user lass (lass@lassul.us).
The masterkey and the admin's initial password come from the zitadel clan
vars generator, the password via an extra steps file so it never hits the
store.
Postgres via peer auth over the local socket. ensureDatabases/ensureUsers
create the database and owner role, so the zitadel role gets no
CREATEROLE/CREATEDB (on PG15 CREATEROLE is close to superuser). That needs
`init zitadel` + start-from-setup instead of the module's start-from-init,
whose init always tries CREATE USER/CREATE DATABASE and fails on the
permission check before the "already exists" check.
nginx terminates TLS and talks h2c to Zitadel via grpc_pass, as Zitadel's
reverse-proxy docs require for its gRPC APIs. Machine identification by
hostname, since the default private-IP/GCP-webhook probes don't fit
neoprism.
The database is pg_dumped by the borg preHook; gatus checks
/debug/healthz. Zitadel 2.71 does not work with PostgreSQL 18 (migration
34_add_cache_schema creates unlogged partitioned tables), neoprism runs 15.
For when.lassul.us (timeful) to show lass's availability without event
details, at https://cal.lassul.us/busy/<token>.ics.
With --busy-token-file, GET /busy/<token>.ics returns the busy time book
itself uses for free slots (CalDAV.busy plus held requests), so the feed
and the booking page agree on what blocks time: recurrences expanded,
all-day events from local midnight, TRANSPARENT and CANCELLED events
left out, TENTATIVE events (own ones and pending booking requests) kept
as busy, since book holds those slots too.
Intervals are merged so overlapping events cannot be counted, and each
VEVENT carries only DTSTART/DTEND in UTC, DTSTAMP (required by RFC
5545), SUMMARY "Busy", TRANSP OPAQUE and a UID hashed from its times.
The window runs from 7 days back to 90 days ahead: book's own 28 day
horizon is too short for polls planned weeks ahead, and the past week
keeps polls that are still open showing sensible data.
The feed is generated per request, so it is as fresh as the calendar;
an unreadable calendar gives 503, never an empty feed. Any other path
under /busy/ gives 404; the token is compared in constant time.
configs/book.nix: the token comes from the prompt-less vars generator
busy-feed and reaches book as a systemd credential, so it never lands in
the Nix store; nginx proxies all of cal.lassul.us/busy/ to book.
book.lassul.us returns 404 for /busy/ so the feed has a single URL.
In the benchmark run (PR 68) the neoprism job spent 29 s between
fetching lazy-store and mounting: the dry-run for the paths to
substitute (13 s in the shared job for the same machine) and their
narinfo closure (the rest, 2,878 paths), both after the 21 s hestia
prefetch of the drvs, which the dry-run needs.
The eval job has the drvs and dry-runs the machines anyway for the
shared derivations; it now keeps each machine's "will be fetched" list
and uploads it with the shared ones. The machine job collects the
closure of that list in the background while hestia prefetches the
drvs, then mounts. Computing the closure in the eval job too would only
move its time onto the eval job, before every machine job starts.
For a single machine the eval job now dry-runs it (about 13 s), against
about 29 s less in the machine job; with more machines the dry-runs ran
already, and in parallel.
The shared job dry-ran every machine to find the derivations needed by
more than one, after a runner start and a prefetch of all drv closures
from hestia. In the benchmark run (PR 68, only neoprism uncached) it
built nothing and still cost the machine job 61 s: 5 s queueing, 6 s
setup, 21 s prefetch (mostly the 47 MiB nixpkgs source), 13 s dry-run,
then 6 s queueing of the machine job.
The eval job has the drvs already, so it dry-runs the machines itself,
in parallel, and only for two or more (one machine shares nothing). It
hands the list to the shared job as an artifact; with an empty list the
shared job is skipped, and the machine jobs wait for the manifest
version of the eval job instead of the shared job's.
`nix run nixpkgs#nix-eval-jobs` resolves nixpkgs through the global
registry (nixpkgs-unstable): a second nixpkgs, fetched as a GitHub
tarball and unpacked into the git cache on every eval job, next to the
locked one the flake evaluation substitutes from cache.nixos.org
anyway. A cold nixpkgs tarball fetch took 18.6 s locally; on the
runner it hides inside the 99 s matrix step of the benchmark run
(PR 68), which prints no progress. With --inputs-from . the eval job
builds nix-eval-jobs from the locked nixpkgs (substituted from
cache.nixos.org) in the tools step and hands the binary to the matrix
step.
The worker count stays at 4: locally nix-eval-jobs over ciToplevels
took 101 s with 4 workers (1.8 cores busy), 72-83 s with 8 and 137 s
with 6 x 2 GiB, too noisy on a shared 16-core host to say anything
about the 4-vCPU runner.
audio-matrix (./matrix.py) watches PipeWire (pw-dump --monitor) and
publishes per source and room an MQTT discovery switch
switch.audio_route_<source>_<room> (attributes source, speaker, kind),
grouped as one HA device per source. Sources are AirPlay (shairport's
"Shairport Sync" stream), LAN (roc-lan) and every connected Bluetooth
phone (bluez_input.*, keyed by MAC, named by its alias); a phone's switches
are removed when it disconnects. The daemon links the source node directly
to roc-<room> with pw-link and re-applies the links whenever nodes come
back (phone reconnects, PipeWire restarts).
The routes are kept per source key in /var/lib/audio-matrix/routes.json, so
a reconnecting phone gets its previous routing. A source never seen before
plays on all rooms: a freshly paired phone should just make sound.
The hub null sink, the "Audio room" select, its buttons, the audio-room
service and command are gone. WirePlumber must not link the sources itself
any more: node.autoconnect = false for bluez_input.* (monitor.bluez.rules
run after name-node.lua sets it true) and roc-lan (source.props), and
PIPEWIRE_AUTOCONNECT=false for shairport-sync, whose pipewire backend
always connects with PW_STREAM_FLAG_AUTOCONNECT. Groups without the null
sink are driven by the phone's node or PipeWire's Dummy-Driver.
The old discovery configs stay retained on mosquitto; clear them once on
acheron:
for t in homeassistant/select/audio_room/config \
homeassistant/button/audio_room_{shodan,icarus,both,off}/config \
gg23/audio-room/set gg23/audio-room/state; do
mosquitto_pub -h localhost -u gg23 -P gg23-mqtt -r -n -t "$t"
done
A Pixel 9a with HD audio negotiated a high-quality codec, and WirePlumber
failed to create the bluez_input node ("mod.adapter: can't get format:
Invalid argument"), so no audio reached the hub and the phone retried
every second. With SBC it works. AAC decoding is available: the
nixpkgs pipewire bluez5 aac plugin links fdk-aac. sbc_xq shares the
sbc endpoint and has a decoder.
sync-containers3 containers (radio, yellow) get the host's
/nix/var/nix/db bind-mounted read-only, so nixos-upgrade failed every
night with "opening lock file /nix/var/nix/db/big-lock: Read-only file
system". Talk to the bind-mounted daemon socket instead, like the
autoswitch service does. Only for boot.isContainer, so other machines
are unaffected by that part.
The guard reported "cannot reach" for any nix flake metadata failure,
including this one; say what actually happened instead. nix's own
error is already on stderr above it.
Load the hermes-voice plugin into the gateway (matrix_dm_send, end_call,
/assist), and enable the gateway's webhook platform on 127.0.0.1:8644
with a deliver_only route to the Matrix home room, through which
hermes-voice asks approvals that no call can hear. The route is signed
with WEBHOOK_SECRET, which hermes-env fills with the API server key that
hermes-voice already holds: no new secret.
gpt-live-1 with client delegation has no client-side function tools: the
only model-to-page signal is session.delegation.created, which carries no
name or arguments. So the voice model does not hang up itself: goodbye,
"that's all", "hang up" and "I'm leaving" are delegated like any request,
and Hermes decides. BACKEND_NOTE tells it to call end_call once requested
work has been started or handed off, never while it still needs an answer
such as an approval.
end_call lives in the gateway plugin. Plugin handlers get the calling
session_id (model_tools.py passes it to registry.dispatch), and the
session context names the platform. The tool succeeds only on the API
server in a session whose lineage starts at a voice-* id: context
compression continues a session under a new id with the old one as
parent_session_id, so the prefix alone would not survive a long call.
Elsewhere it returns an error and changes nothing.
hermes-voice sees the tool in the run's events (tool.completed of
end_call without error) and passes an end_call item to the page instead
of a tool item. The page hands Hermes's reply to the voice model as
usual and closes the session once the voice has spoken it and been quiet
for 800 ms, at the latest after 30 s; close() now runs only once. A run
no page listens to (hang-up flush, page gone) only logs it.
Hermes registers no agent-callable send_message tool on any platform
(tools/send_message_tool.py: "intentionally NOT registered as an
agent-callable model tool"; toolsets.py: "deliberately no agent-callable
send_message tool"), so there is no config switch, platform toolsets
included, that lets voice sessions on the API server write to Matrix. The
plugin adds matrix_dm_send, limited to the Matrix home room (the owner's
DM) and sent through the gateway's live adapter by Hermes's own send
helper, which also mirrors the message into the DM's session. Reading the
DM needs nothing new: session_search (chat_history_lookup) already reads
the Matrix session's transcript from any session.
/assist approve|deny [code] answers an approval hermes-voice asked in the
DM. A Matrix /approve cannot do it: it only resolves approvals of the
chat's own session, and API runs are keyed by run id. /voice is taken by
Hermes itself.
Hermes does not stop a run when the /v1/runs/{id}/events subscriber goes
away (the SSE handler only detaches it), so runs already survived the
page. But nothing listened any more: an approval request then waited for
nobody and was denied by Hermes's timeout. hermes-voice now follows every
run it starts in its own thread, page or no page; the page replays the
run's items and gets blank keepalive lines so a vanished page is noticed.
On hang-up (button, idle, page close) the page posts /hangup with
keepalive: runs of the conversation count as unheard from then on, and
what the user said since the last delegation becomes a run of its own
with a note that nobody listens. Lines that the model's delegation
already took are not sent again, and a delegation still in its settle
delay leaves them to the hang-up. Only a correction within the same call
(replace) stops the previous run; a new call no longer stops work an
earlier call started.
An approval no call can answer is posted to the owner's Matrix DM (the
gateway's Matrix home room) through the gateway's webhook platform
(--matrix-webhook, signed with the API server key) with a short code;
/approve takes that code, from the page or from the Matrix side with the
API server key.
Approvals also failed outright before: under the gateway's session
authority the approval request is named prompt_id and the answer must be
exactly {request_id, execution_generation, choice}, while hermes-voice
sent only {choice} (invalid_params).
The persona and the server's texts name no particular user: they speak
of "the user" and of the owner's Matrix DM.
With the lazy /nix/store in the machine jobs, hestia's drain read every
cache.lassul.us-only input path to mirror it into the GitHub Actions
cache, downloading 375-760 MiB per job through the lazy store and
spending 1-4.5 min uploading to the Actions cache at 2-11 MiB/s (PR 65
run). Those paths are already in cache.lassul.us; niks3 uploads
everything newly built.
upstream-cache-key-names replaces hestia's default list, so it repeats
cache.nixos.org-1 next to cache.lassul.us-1. In the eval job,
filter-drv-closures applies the same keys to the drv closures; members
signed by cache.lassul.us are substituted from there by the later jobs.
The day side now shows what the geostationary satellites see, like
zoom.earth, instead of the base map under white clouds synthesised from
infrared:
- Europe/Africa/Atlantic: MTG (Meteosat-12) rgb_truecolour from EUMETSAT.
rgb_geocolour looks the same by day but renders ~4x slower on the
server and timed out (502) for full 2048 px tiles.
- Americas/Pacific: GOES-East and GOES-West GeoColor from GIBS.
- Asia/Australia: GIBS has no Himawari colour product. Himawari band 3
(red, 10 min) is turned into white cloud over the base map, with the
cloud opacity being the reflectance beyond the ground's own albedo.
The daily VIIRS true colour mosaics were the alternative, but their
clouds are up to a day old, have swath seams of different times and
would not match the IR clouds next to them.
Each source is weighted by how well its satellite sees the point: full
trust up to 62° view zenith, none from 78° (the limb is smeared), and
the more nadir satellite wins in overlaps with cos^8 of the zenith
angle, so seams are soft. Trust also needs the sun not to be in front
of the satellite: looking towards a low sun both MTG (mornings over the
Indian Ocean) and Himawari (evenings) are washed out to a pale band
with straight edges, while a long path with the sun behind the
satellite is fine (Berlin at winter noon). A plain path-length limit
would have dropped the latter. Near the terminator (cosz < 0.2)
GeoColor fades into its purple night blend, so the live imagery fades
out there and the IR clouds over the base map take over as before.
Where it is shown, the synthetic clouds and so their relief shading and
shadows are scaled down, so the real image is not shaded twice.
All live layers (including the IR clouds) now request an explicit TIME,
the render time floored to 10 min: without TIME GIBS served tiles of
one layer from different frames (sun glint cut at a tile edge), and it
makes --time render the imagery of that time, not only the sun. The
TIME is capped at the newest frame the server lists (GIBS
DescribeDomains, EUMETSAT per-layer capabilities), since GIBS answers
an unpublished time with an empty image and EUMETSAT with an error;
MTG true colour stops producing frames while its disk is dark.
GIBS also now and then answers with an empty 1 kB image for a frame it
has (the same request succeeds when repeated), which left a quarter of
the Himawari disk missing, so empty tiles are asked for up to 3 times.
A machine job substituted the whole input closure of the derivations it
builds, e.g. 3.4 GiB for neoprism, of which the builds open a few dozen
MiB. .github/lazy-store.sh now runs before the build: it takes what
`nix build --dry-run` would fetch, collects its closure from the
narinfos of cache.nixos.org and cache.lassul.us (tools.lazy-store),
mounts lazy-store on /nix/store, registers the paths with
`nix-store --load-db` and copies their signatures. The build itself,
ci-build.sh's niks3 push and hestia stay as they were: outputs are
written to the real store dir below the mount, and ci-build.sh's
before/after diff of the store starts after the registration.
If anything fails before the paths are registered, it logs an error,
undoes the mount and exits 0, so the build substitutes eagerly. The FUSE
log and the fetched NARs are printed by an always-run step after the
build.
The eval job builds lazy-store next to niks3 and pushes it, the machine
jobs substitute it. The eval and shared jobs keep the plain store: eval
builds nothing but niks3 and lazy-store, and shared builds package
derivations, which read most of their inputs anyway.
FUSE file system for CI's machine builds, ported from the experiment on
ci-lazy-fuse (.github/exp/lazy-store). It mounts on /nix/store the real
store dir (all writes go there) merged with store paths of
cache.nixos.org and cache.lassul.us, whose metadata comes from their
.ls listings and whose NAR is fetched, checked and unpacked on the first
open() of a file in them; reads go through kernel FUSE passthrough. In
the experiment, rebuilding neoprism's system-units, etc, system-path and
toplevel downloaded 62 MiB instead of the 3,455 MiB of an eager
substitution.
Changes from the experiment:
- closure takes the store paths to make valid on stdin (what
`nix build --dry-run` would fetch) instead of walking derivations, and
leaves out paths whose closure is not entirely in the caches, so that
nix never sees a valid path with a missing reference; nix substitutes
those as usual.
- closure also writes the fetched narinfos as a file:// binary cache:
`nix-store --load-db` registers no signatures, and niks3's
--ignore-upstream-key and hestia's upstream-cache-filter go by the
signatures in the local database, so without them both would upload
(and thereby download) every cache.nixos.org path in an output's
closure. `nix store copy-sigs` adds them from there.
- mount fails if a path has no listing instead of leaving it out, as
every path gets registered.
Static, so that once it serves /nix/store it loads nothing from there.
A flake package, so CI builds it once and pushes it to cache.lassul.us,
and the jobs substitute it. passthru.tests.fetch covers fetch, closure
and the registration with signatures against a file:// cache.
The GIBS VIIRS_NOAA20_Thermal_Anomalies_375m_All raster shows each
detection as a coarse blob, far bigger than its 375 m footprint at
wallpaper scale, so the fire layer was loud and distracting.
Fetch FIRMS' keyless global 24 h VIIRS CSVs of NOAA-20, NOAA-21 and
Suomi NPP instead (cached, refreshed hourly like the old layer), drop
low-confidence detections and draw each one as an antialiased dot in
screen space through the same forward mapping as the markers: about
1 px at 1080p, growing slightly with the lens zoom, opacity from FRP on
a log scale between 1 and 500 MW. The colour and max blend under the
clouds stay as before.
Mercator at 16:9 only showed about +-71 degrees, cutting off northern
Greenland and the Arctic. The new variant world is plate carrée, north up,
centred on the equator and the home longitude, showing -90..90 degrees
stretched vertically to fill the screen. It replaces mercator in the CLI
--projection choices and the server's variant parameter, without an alias
for the old name. lens and plain are unchanged.
acheron gets BlueZ (adapter "gg23", UB500 dongle) as an A2DP sink only;
WirePlumber's bluez monitor targets received streams at the "hub" sink,
like roc-lan and AirPlay. A bt-agent (NoInputNoOutput) accepts pairing
without PIN. A Home Assistant button (MQTT discovery) opens a 120s
discoverable/pairable window; afterwards bonded devices are trusted so they
reconnect any time, and a binary_sensor shows the window state.
Replaces the farm call. eval evaluates the new ciToplevels output once
with hestia's matrix subaction (the same machines the farm built: all
nixosConfigurations), shared builds the derivations needed by more than
one uncached machine once and hands the hestia manifest version with
their outputs to the machine jobs, one job per uncached machine builds
its toplevel by drv path. The ci job stays the required check; shared
and the machine jobs are skipped when nothing is uncached.
Every job pushes the store paths it added, built or substituted from
hestia, to niks3 with its GitHub OIDC token, as the farm did, so
deploys find the toplevels in cache.lassul.us. The niks3 client is the
new packages.niks3 output.
Measured on the ci-hestia-shared experiment with a nixpkgs bump: 15 min
(eval 3.3, shared 3.3, 1-4 per machine) instead of 49 with the farm. No
more queueing of runs for the farm.
MCP server that controls the everyday Firefox through a WebExtension and
a native messaging host instead of Marionette/WebDriver, so
navigator.webdriver stays false and Cloudflare/reCAPTCHA stop flagging
normal browsing. No browser restart is needed to attach, tabs are
addressed by id on demand.
barnacle installs the server and links the native messaging host
manifest into /Library/Application Support/Mozilla/NativeMessagingHosts.
The extension is signed as unlisted on AMO via browser-mcp-sign.
atuin 18.23.0 fails this e2e test in unsandboxed darwin builds: the test
shell reads nix-darwin's /etc/bashrc, which resets PATH so the atuin
under test is not used. Fixed in nixpkgs by NixOS/nixpkgs#574509
(NOSYSBASHRC in preCheck); drop this package once our nixpkgs pin
includes it. Linux keeps the unmodified nixpkgs derivation.
ntfy.sh's free tier counts every UnifiedPush message against the
subscriber: 250 messages/day shared by all UnifiedPush apps on the phone.
Once used up it answers 429, phonetpm can no longer wake the phone and
every SSH login and deploy fails. Run our own ntfy on neoprism instead,
anonymous like ntfy.sh so app servers publish without accounts.
Rate limits: ntfy 2.29 defaults stay, except for an exemption.
- visitor-message-daily-limit is 0 (unlimited), only the request limiter
applies.
- visitor-subscriber-rate-limiting is off (ntfy.sh turns it on), so
publishes to up* topics are charged to the publisher, not to the phone.
The phone only spends requests on its subscription connections (30
allowed, the app uses one per server).
- The request limit is 60 burst, then one per 5s, per publisher IP.
Synapse on neoprism sends every Matrix push of the phone from one
address; encrypted and bridged rooms notify per message, so bridge
catch-up can exceed that. visitor-request-limit-exempt-hosts exempts
neoprism's own addresses. phonetpm daemons publish one wake per
connection and stay far below the limit.
- behind-proxy: ntfy takes the right-most X-Forwarded-For address, the
one nginx appends, so visitors are not all 127.0.0.1 and a client
cannot spoof its address.
The module enables attachments by default (attachment-cache-dir), set it
empty to keep them off. nginx settings follow ntfy's example config
(buffering off, 3m timeouts against the 45s keepalive, websockets).
No backup: the state dir (/var/lib/private/ntfy-sh) holds the message
cache (12h, disposable) and an auth db without users.
Since d9ed167 networkd installed the route to the container itself.
networkd adds it as soon as ve-riot gets carrier, before the container
start script runs `ip route add 10.233.1.2 dev ve-riot`, so the script
fails with "RTNETLINK answers: File exists" and container@riot ends up
in a restart loop. Before the reboot riot had never restarted since
d9ed167, so nobody noticed.
The script's route and address come from nixpkgs' nixos-containers
module and can't be turned off without patching nixpkgs (localAddress
also sets the container side), so the networkd route has to go. Now
ve-riot is Unmanaged=yes instead, and the start script owns its
address and route. Checked against systemd 261:
- An unmanaged link gets no configuration from networkd, so nothing is
in place before the script's `ip route add` and it succeeds on every
start.
- networkd only drops foreign routes for the link it is configuring
(link_drop_unmanaged_config, CONFIGURING state). When link_drop_routes
marks foreign routes it skips routes on other interfaces, "Otherwise,
routes assigned to unmanaged interfaces will be dropped". So a networkd
restart or reload no longer touches the script's route. That was the
bug d9ed167 fixed.
KeepConfiguration=static could not keep the script's route anyway: it
only protects foreign routes with protocol static, and `ip route add`
uses protocol boot. LinkLocalAddressing=no and KeepConfiguration=static
mean nothing on an unmanaged link, so they are gone, along with the
commented-out IPForward/IPMasquerade lines. Forwarding and NAT stay in
nftables and the sysctl. No other .network file on neoprism matches
ve-riot (99-ethernet-default-dhcp needs Kind=!*), but Unmanaged=yes
also keeps a later catch-all file from taking over ve-riot.
neoprism's NVMe disks report critical_warning 4 and a failing SMART
status only because percentage_used is above 100% (rated endurance
exceeded). Hetzner confirmed with the manufacturer that bit 0x04 is set
for that reason and that the drives are fine as long as available_spare
is above available_spare_threshold, which NvmeSpareLow watches. smartctl
derives smart_status from the same bit, so both alerts stayed red on
worn but healthy drives.
NvmeCriticalWarning now fires only on critical_warning bits other than
0x04 (bitmap_and with 251). SmartFailed skips devices that
smartctl_exporter reports with protocol="NVMe" and keeps covering
SATA/SCSI disks.
smartctl_exporter runs on every non-container machine, bound to
localhost and scraped by Vector into the host-tagged metrics path.
Alerts on a failed SMART status, NVMe critical warnings, new NVMe media
errors and NVMe spare below its threshold.
Memory PSI comes from /proc/pressure/memory via an exec source, as a
counter of stalled seconds, so the alert takes an exact 10m rate instead
of the kernel's 5m avg300 EWMA. Plus an alert on any OOM kill.
MemoryPressure threshold 10%: some task stalling on memory a tenth of
the time over 10 minutes is sustained reclaim/swap thrash, while short
spikes such as big builds average out.