Clone this repository
For self-hosted knots, clone URLs may differ based on your setup.
HedgeDoc looks up users only by profileid, in one table with the GitHub
logins, whose profileid is the numeric GitHub user id. Without
userProfileIdAttr it is the username, which anyone chooses when
registering at id.lassul.us: a Zitadel user named after someone's GitHub
id logged in as that GitHub user, and a renamed or deleted username
handed its pad account to the next user taking the name. sub is the
Zitadel user id, which Zitadel generates and never reuses.
Existing Zitadel accounts in HedgeDoc are keyed by their login name and
need their profileid changed to the Zitadel user id.
B2 behind cache.lassul.us sometimes sends the headers and then no body;
when all retries for one .ls failed, `mount` gave up ("17 of 2892 store
paths without a listing") and the whole build substituted eagerly.
Now a path whose listing can't be fetched (or parsed, or built from its
NAR) is treated like a path not in the caches: it and everything
referring to it (from the references in db.dump) are dropped from
paths.json and db.dump before mounting, so they are neither served nor
registered and Nix substitutes them as usual; the rest stays lazy. Each
is logged once with its error, the counts go to no-listing.json, and
lazy-store.sh prints them in the "mounted" line (and counts the
registered paths after the mount). Narinfo fetch errors already ended up
in `missing` in `closure`.
16 of the 17 paths without a listing in CI run 39052697442 (neoprism)
failed with "missing field `entries`": cache.lassul.us writes empty
directories as {"type":"directory"}, e.g. 3k9vl6iwf58qy65wcjqccqykgzypm7va.ls
is {"version":1,"root":{"type":"directory"}}. harmonia-file-core's
Directory requires "entries", so since 207a89b5 that was an error; the
hand-walked parser before read a missing "entries" as empty. Fill it in
before deserializing. "size" of regular files is always written (also
for empty files).
The 17th, 9pczdaxss3xdmidr8cvgpw8ccn0agnhq.ls, is not a client bug: B2
itself (also queried directly, over HTTP/1.1) now and then answers 200
with Content-Length 78 and closes without sending a body byte. nginx
turns that into an HTTP/2 RST_STREAM INTERNAL_ERROR, which reqwest
reports as "error decoding response body". The same request succeeds
moments later; the existing retries cover it unless all six tries hit
a truncated response, as in that CI run.
closure uses harmonia-store-nar-info's parse_narinfo_txt and
harmonia-store-path's StorePath instead of our key/value narinfo map
and string splitting: the closure walk, referrers and missing sets are
keyed by StorePath (whose order is the string order, so paths.json and
db.dump come out the same), references, deriver, NarHash, NarSize,
URL, Compression and FileSize come from the parsed NarInfo. A narinfo
that doesn't parse counts as a failed fetch from that cache (try the
next one) instead of failing the whole run later on a missing field.
Roots that aren't store paths are an error.
paths.json, db.dump, narinfo/ and closure.json are byte-identical to
before for the closures of hello and git from cache.nixos.org.
The fetch test's path without a narinfo used an "e" hash, which isn't
nix32 and so no store path.
Our NAR reader goes: NARs are parsed by harmonia-file-nar's parse_nar
and written by its restore (async, bounded queue to one writer thread;
it also checks entry names, sorting and padding like Nix). Files come
out with the umask's permissions, so a pass afterwards sets the store's
0444/0555 like our reader did.
The NAR hash is computed with harmonia-utils-hash (aws-lc sha256) in a
tokio-util InspectReader instead of our HashingReader and sha2, and
narinfo NarHashes are parsed with harmonia (any encoding with the
sha256: prefix) instead of comparing base32/base16 strings. Data after
the NAR counts into the size, so the NarSize check still rejects it.
The .ls listings are harmonia-file-core FileTrees now instead of our
Tree enum converted from them. The test unpacks a NAR made by
harmonia's NarByteStream instead of a hand-written one.
The I/O runs on a tokio runtime now:
- HTTP via reqwest (rustls on aws-lc, HTTP/2, bundled webpki roots as
before) instead of ureq. Instead of our own caching ureq::Resolver the
cache hosts are resolved once at startup and pinned with
resolve_to_addrs, which keeps getaddrinfo() (and NSS modules from the
store) out of requests once /nix/store is mounted. The retry loop
stays: reqwest-middleware/retry would not save code, as body read
errors are retried too.
- Decompression via async-compression (zstd, xz, bzip2, gzip, brotli,
concatenated members like Nix) instead of zstd/xz2/bzip2/flate2/
brotli-decompressor; the magic sniffing for narinfo/.ls stays.
- The closure walk and the .ls listings run with buffer_unordered
instead of par_map threads.
- FUSE workers stay threads; open() fetches via Handle::block_on, and
concurrent opens of a path wait on a tokio Mutex.
- The compressed size is counted with tokio-util's InspectReader instead
of our CountingReader. The NAR is still unpacked by our reader for now,
through SyncIoBridge.
Replace our nix32 encoder with harmonia-utils-base-encoding's base32 and
the hand-walked serde_json::Value of the caches' .ls listings with
harmonia-file-core's FileTree, which is the serde model of Nix's file
system object JSON. Both are pure crates (data-encoding/derive_more,
serde/serde_json/thiserror), pulled as a git dependency pinned to
harmonia 3edb37ae: only harmonia-utils-hash and harmonia-store-core exist
on crates.io, as 0.0.0-alpha.0 placeholders. A listing node without
"size" (regular) or "entries" (directory) is now an error instead of
being read as 0 / empty; Nix always writes both.
Kept ours:
- NAR unpacking: harmonia-file-nar is a tokio AsyncRead parser/restorer;
we unpack synchronously from the blocking HTTP/decompressor stream
while hashing, in a sync FUSE daemon.
- narinfo parsing, store path names and NarHash parsing:
harmonia-store-nar-info, -store-path and -utils-hash all pull in
aws-lc-rs (aws-lc-sys C build), blake3 and md5, nar-info also
ed25519-dalek and getrandom: +43 crates for ~15 lines. closure also
passes the narinfo text through verbatim, and harmonia's parser
rejects whole narinfos for fields we don't use (Sig, CA, FileHash).
- The in-memory Tree: matching FileTree(FileSystemObject::Regular(
Regular { .. })) in lazy.rs is clumsier than our flat enum.