tools.lazy-store: tokio and reqwest instead of ureq and threads
The I/O runs on a tokio runtime now:
- HTTP via reqwest (rustls on aws-lc, HTTP/2, bundled webpki roots as
before) instead of ureq. Instead of our own caching ureq::Resolver the
cache hosts are resolved once at startup and pinned with
resolve_to_addrs, which keeps getaddrinfo() (and NSS modules from the
store) out of requests once /nix/store is mounted. The retry loop
stays: reqwest-middleware/retry would not save code, as body read
errors are retried too.
- Decompression via async-compression (zstd, xz, bzip2, gzip, brotli,
concatenated members like Nix) instead of zstd/xz2/bzip2/flate2/
brotli-decompressor; the magic sniffing for narinfo/.ls stays.
- The closure walk and the .ls listings run with buffer_unordered
instead of par_map threads.
- FUSE workers stay threads; open() fetches via Handle::block_on, and
concurrent opens of a path wait on a tokio Mutex.
- The compressed size is counted with tokio-util's InspectReader instead
of our CountingReader. The NAR is still unpacked by our reader for now,
through SyncIoBridge.
tools.lazy-store: tokio and reqwest instead of ureq and threads
The I/O runs on a tokio runtime now:
- HTTP via reqwest (rustls on aws-lc, HTTP/2, bundled webpki roots as
before) instead of ureq. Instead of our own caching ureq::Resolver the
cache hosts are resolved once at startup and pinned with
resolve_to_addrs, which keeps getaddrinfo() (and NSS modules from the
store) out of requests once /nix/store is mounted. The retry loop
stays: reqwest-middleware/retry would not save code, as body read
errors are retried too.
- Decompression via async-compression (zstd, xz, bzip2, gzip, brotli,
concatenated members like Nix) instead of zstd/xz2/bzip2/flate2/
brotli-decompressor; the magic sniffing for narinfo/.ls stays.
- The closure walk and the .ls listings run with buffer_unordered
instead of par_map threads.
- FUSE workers stay threads; open() fetches via Handle::block_on, and
concurrent opens of a path wait on a tokio Mutex.
- The compressed size is counted with tokio-util's InspectReader instead
of our CountingReader. The NAR is still unpacked by our reader for now,
through SyncIoBridge.
tools.lazy-store: tokio and reqwest instead of ureq and threads
The I/O runs on a tokio runtime now:
- HTTP via reqwest (rustls on aws-lc, HTTP/2, bundled webpki roots as
before) instead of ureq. Instead of our own caching ureq::Resolver the
cache hosts are resolved once at startup and pinned with
resolve_to_addrs, which keeps getaddrinfo() (and NSS modules from the
store) out of requests once /nix/store is mounted. The retry loop
stays: reqwest-middleware/retry would not save code, as body read
errors are retried too.
- Decompression via async-compression (zstd, xz, bzip2, gzip, brotli,
concatenated members like Nix) instead of zstd/xz2/bzip2/flate2/
brotli-decompressor; the magic sniffing for narinfo/.ls stays.
- The closure walk and the .ls listings run with buffer_unordered
instead of par_map threads.
- FUSE workers stay threads; open() fetches via Handle::block_on, and
concurrent opens of a path wait on a tokio Mutex.
- The compressed size is counted with tokio-util's InspectReader instead
of our CountingReader. The NAR is still unpacked by our reader for now,
through SyncIoBridge.
tools.lazy-store: init
FUSE file system for CI's machine builds, ported from the experiment on
ci-lazy-fuse (.github/exp/lazy-store). It mounts on /nix/store the real
store dir (all writes go there) merged with store paths of
cache.nixos.org and cache.lassul.us, whose metadata comes from their
.ls listings and whose NAR is fetched, checked and unpacked on the first
open() of a file in them; reads go through kernel FUSE passthrough. In
the experiment, rebuilding neoprism's system-units, etc, system-path and
toplevel downloaded 62 MiB instead of the 3,455 MiB of an eager
substitution.
Changes from the experiment:
- closure takes the store paths to make valid on stdin (what
`nix build --dry-run` would fetch) instead of walking derivations, and
leaves out paths whose closure is not entirely in the caches, so that
nix never sees a valid path with a missing reference; nix substitutes
those as usual.
- closure also writes the fetched narinfos as a file:// binary cache:
`nix-store --load-db` registers no signatures, and niks3's
--ignore-upstream-key and hestia's upstream-cache-filter go by the
signatures in the local database, so without them both would upload
(and thereby download) every cache.nixos.org path in an output's
closure. `nix store copy-sigs` adds them from there.
- mount fails if a path has no listing instead of leaving it out, as
every path gets registered.
Static, so that once it serves /nix/store it loads nothing from there.
A flake package, so CI builds it once and pushes it to cache.lassul.us,
and the jobs substitute it. passthru.tests.fetch covers fetch, closure
and the registration with signatures against a file:// cache.
tools.lazy-store: use harmonia for nixbase32 and the .ls listing format
Replace our nix32 encoder with harmonia-utils-base-encoding's base32 and
the hand-walked serde_json::Value of the caches' .ls listings with
harmonia-file-core's FileTree, which is the serde model of Nix's file
system object JSON. Both are pure crates (data-encoding/derive_more,
serde/serde_json/thiserror), pulled as a git dependency pinned to
harmonia 3edb37ae: only harmonia-utils-hash and harmonia-store-core exist
on crates.io, as 0.0.0-alpha.0 placeholders. A listing node without
"size" (regular) or "entries" (directory) is now an error instead of
being read as 0 / empty; Nix always writes both.
Kept ours:
- NAR unpacking: harmonia-file-nar is a tokio AsyncRead parser/restorer;
we unpack synchronously from the blocking HTTP/decompressor stream
while hashing, in a sync FUSE daemon.
- narinfo parsing, store path names and NarHash parsing:
harmonia-store-nar-info, -store-path and -utils-hash all pull in
aws-lc-rs (aws-lc-sys C build), blake3 and md5, nar-info also
ed25519-dalek and getrandom: +43 crates for ~15 lines. closure also
passes the narinfo text through verbatim, and harmonia's parser
rejects whole narinfos for fields we don't use (Sig, CA, FileHash).
- The in-memory Tree: matching FileTree(FileSystemObject::Regular(
Regular { .. })) in lazy.rs is clumsier than our flat enum.