This repository has no description
.github.ci: build the NixOS machines on plain runners with hestia
Replaces the farm call. eval evaluates the new ciToplevels output once
with hestia's matrix subaction (the same machines the farm built: all
nixosConfigurations), shared builds the derivations needed by more than
one uncached machine once and hands the hestia manifest version with
their outputs to the machine jobs, one job per uncached machine builds
its toplevel by drv path. The ci job stays the required check; shared
and the machine jobs are skipped when nothing is uncached.
Every job pushes the store paths it added, built or substituted from
hestia, to niks3 with its GitHub OIDC token, as the farm did, so
deploys find the toplevels in cache.lassul.us. The niks3 client is the
new packages.niks3 output.
Measured on the ci-hestia-shared experiment with a nixpkgs bump: 15 min
(eval 3.3, shared 3.3, 1-4 per machine) instead of 49 with the farm. No
more queueing of runs for the farm.
author
Hermes
committer
lassulus
date
(Oct 16, 2026, 10:39 PM CEST)
commit
5995061c
5995061c7c70745f5736fe401c38163558d6686e
parent
662ea3fa
662ea3fac6651c79f4e5cf5447d64d9179cfece0
+238
-21
+4
-2
AGENTS.md
+4
-2
AGENTS.md
···
93
93
```
94
94
95
95
`--rebase` keeps the individual commits on master. Only the user (repo admin)
96
-
bypasses the ruleset. The `ci` check includes a farm build of all machines
97
-
(farm.yml, called from ci.yml). The update-pkgs bot's PRs go through
96
+
bypasses the ruleset. The `ci` check includes the build of all NixOS
97
+
machines on GitHub runners, with hestia (GitHub Actions cache) as binary
98
+
cache; every CI build is pushed to cache.lassul.us (niks3), so deploys
99
+
substitute the toplevels from there. The update-pkgs bot's PRs go through
98
100
the ruleset as well: they run ci.yml like any PR and merge via auto-merge.
99
101
100
102
## Machine Facts and Secrets via vars
+8
-1
flake.nix
+8
-1
flake.nix
···
228
228
map (dir: ./tools + "/${dir}/flake-module.nix") toolDirs
229
229
);
230
230
flake.nixosConfigurations = clan.config.nixosConfigurations;
231
+
# The machine toplevels as plain derivations, for CI's build matrix
232
+
# (nix-eval-jobs, .github/workflows/ci.yml).
233
+
flake.ciToplevels = nixpkgs.lib.mapAttrs (
234
+
_: c: c.config.system.build.toplevel
235
+
) clan.config.nixosConfigurations;
231
236
flake.clanInternals = clan.config.clanInternals;
232
237
flake.darwinConfigurations = clan.config.darwinConfigurations;
233
238
flake.clan = clan.config;
···
244
249
}:
245
250
{
246
251
packages.clan-cli = clan-core.packages.${system}.clan-cli;
252
+
# CI pushes its builds to cache.lassul.us with it (.github/niks3-push.sh)
253
+
packages.niks3 = inputs.niks3.packages.${system}.niks3;
247
254
devShells.default = pkgs.mkShell {
248
255
packages = [
249
256
self.packages.${pkgs.system}.pass
···
258
265
inputs.llm-agents.packages.${system}
259
266
// lib.optionalAttrs (self.packages.${system} ? orca) { inherit (self.packages.${system}) orca; };
260
267
# Check Darwin configurations can be evaluated (without building).
261
-
# The NixOS machines are built by the farm (.github/workflows/farm.yml).
268
+
# The NixOS machines are built by .github/workflows/ci.yml (ciToplevels).
262
269
checks = lib.mapAttrs' (
263
270
name: config:
264
271
lib.nameValuePair "darwin-${name}" (
+16
.github/ci-build.sh
+16
.github/ci-build.sh
···
1
+
#!/usr/bin/env bash
2
+
# Builds INSTALLABLE... for ci.yml and pushes every store path the build
3
+
# added, built or substituted from hestia, to cache.lassul.us
4
+
# (niks3-push.sh), also when some of them failed.
5
+
# Usage: ci-build.sh INSTALLABLE...
6
+
set -euo pipefail
7
+
before=$(mktemp)
8
+
added=$(mktemp)
9
+
nix path-info --all | sort >"$before"
10
+
status=0
11
+
nix build -L --no-link --keep-going "$@" || status=$?
12
+
nix path-info --all | sort | comm -13 "$before" - | { grep -v '\.drv$' || true; } >"$added"
13
+
if [ -s "$added" ]; then
14
+
"$(dirname "$(realpath "$0")")/niks3-push.sh" --stdin <"$added"
15
+
fi
16
+
exit "$status"
+21
.github/niks3-push.sh
+21
.github/niks3-push.sh
···
1
+
#!/usr/bin/env bash
2
+
# Pushes the closures of PATH... to niks3.lassul.us, served as
3
+
# cache.lassul.us, with the job's GitHub OIDC token (needs id-token: write;
4
+
# niks3 on neoprism trusts tokens of lassulus/superconfig, see
5
+
# configs/nix-farm/cache.nix). cache.nixos.org keeps serving what it signed, those
6
+
# paths are left out. Needs niks3 in PATH.
7
+
# Usage: niks3-push.sh PATH...
8
+
# niks3-push.sh --stdin the paths from stdin
9
+
# niks3-push.sh --token the token, as niks3's --auth-token-script
10
+
set -euo pipefail
11
+
url=https://niks3.lassul.us
12
+
13
+
if [ "${1:-}" = --token ]; then
14
+
curl -fsS --retry 4 --retry-all-errors \
15
+
-H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN:?needs id-token: write}" \
16
+
"$ACTIONS_ID_TOKEN_REQUEST_URL&audience=$url" | jq -ce '{ token: .value }'
17
+
exit
18
+
fi
19
+
20
+
niks3 push --server-url "$url" --auth-token-script "$(realpath "$0") --token" \
21
+
--ignore-upstream-key cache.nixos.org-1 "$@"
+149
-16
.github/workflows/ci.yml
+149
-16
.github/workflows/ci.yml
···
6
6
pull_request:
7
7
workflow_dispatch:
8
8
9
-
# A newer run for the same ref replaces a running one, its farm included.
10
-
# Runs for different refs queue for the farm, see the farm job.
9
+
# A newer run for the same ref replaces a running one.
11
10
concurrency:
12
11
group: ci-${{ github.ref }}
13
12
cancel-in-progress: true
14
13
15
-
# One job per check shows failing checks in the run overview. The NixOS
16
-
# machines are built by the farm, one job per machine.
14
+
# One job per check shows failing checks in the run overview, the same for
15
+
# the NixOS machines, see the eval job.
17
16
jobs:
18
17
list:
19
18
runs-on: ubuntu-latest
···
68
67
nix eval --raw ".#$output.x86_64-linux" --apply 'xs: builtins.deepSeq (builtins.mapAttrs (_: x: x.drvPath) xs) "${toString (builtins.length (builtins.attrNames xs))} evaluated\n"'
69
68
done
70
69
71
-
# Builds all NixOS machines (farm.yml). Its iroh endpoints are fixed, so one
72
-
# farm at a time across all runs; the others wait FIFO. Without `queue: max`
73
-
# GitHub keeps one pending run per group and cancels the one before.
74
-
farm:
75
-
uses: ./.github/workflows/farm.yml
76
-
secrets: inherit
70
+
# The NixOS machines, built on plain runners with hestia (the GitHub Actions
71
+
# cache) as binary cache next to cache.lassul.us:
72
+
#
73
+
# - eval: evaluates ciToplevels once, uploads the drv closures to hestia
74
+
# and lists the machines whose toplevel is in neither cache.
75
+
# - shared: builds the derivations needed by more than one of them once,
76
+
# so the machine jobs substitute them instead of each building them.
77
+
# - one job per listed machine, builds its toplevel by drv path.
78
+
#
79
+
# Every job pushes what it added to the store to cache.lassul.us
80
+
# (.github/ci-build.sh), for deploys.
81
+
eval:
82
+
runs-on: ubuntu-latest
77
83
permissions:
78
84
contents: read
85
+
actions: read # hestia: detect evicted cache entries upfront
79
86
id-token: write # niks3 push
80
-
actions: read # the jobs watch each other
81
-
concurrency:
82
-
group: farm
83
-
queue: max
87
+
outputs:
88
+
matrix: ${{ steps.matrix.outputs.matrix }}
89
+
any-jobs: ${{ steps.matrix.outputs.any-jobs }}
90
+
manifest-version: ${{ steps.matrix.outputs.manifest-version }}
91
+
niks3: ${{ steps.niks3.outputs.path }}
92
+
steps:
93
+
- uses: actions/checkout@v4
94
+
- uses: cachix/install-nix-action@v31
95
+
with:
96
+
extra_nix_config: |
97
+
trusted-users = root runner
98
+
extra-substituters = https://cache.lassul.us
99
+
extra-trusted-public-keys = cache.lassul.us-1:aOMEO5TU6eLvhyPi0jmgjR2sVlGjMq3rK0MeACjTcQ0=
100
+
- uses: Mic92/hestia@v3
101
+
with:
102
+
upstream-cache-filter: true
103
+
filter-drv-closures: true
104
+
# the niks3 client of all jobs, the others substitute it
105
+
- id: niks3
106
+
run: |
107
+
niks3=$(nix build --no-link --print-out-paths .#niks3)
108
+
echo "path=$niks3" >> "$GITHUB_OUTPUT"
109
+
PATH=$niks3/bin:$PATH .github/niks3-push.sh "$niks3"
110
+
- id: matrix
111
+
uses: Mic92/hestia/matrix@v3
112
+
with:
113
+
flake: ".#ciToplevels"
114
+
nix-eval-jobs: "nix run nixpkgs#nix-eval-jobs -- --workers 4 --max-memory-size 3072"
115
+
- env:
116
+
MATRIX: ${{ steps.matrix.outputs.matrix }}
117
+
run: |
118
+
echo "not cached: $(jq -r '[.include[].name] | join(" ")' <<<"$MATRIX")" |
119
+
tee -a "$GITHUB_STEP_SUMMARY"
120
+
121
+
shared:
122
+
needs: eval
123
+
if: needs.eval.outputs.any-jobs == 'true'
124
+
runs-on: ubuntu-latest
125
+
permissions:
126
+
contents: read
127
+
actions: read # hestia: detect evicted cache entries upfront
128
+
id-token: write # niks3 push
129
+
outputs:
130
+
manifest-version: ${{ steps.build.outputs.manifest-version }}
131
+
steps:
132
+
- uses: actions/checkout@v4
133
+
- uses: cachix/install-nix-action@v31
134
+
with:
135
+
extra_nix_config: |
136
+
trusted-users = root runner
137
+
extra-substituters = https://cache.lassul.us
138
+
extra-trusted-public-keys = cache.lassul.us-1:aOMEO5TU6eLvhyPi0jmgjR2sVlGjMq3rK0MeACjTcQ0=
139
+
- uses: Mic92/hestia@v3
140
+
with:
141
+
upstream-cache-filter: true
142
+
wait-manifest-version: ${{ needs.eval.outputs.manifest-version }}
143
+
- env:
144
+
NIKS3: ${{ needs.eval.outputs.niks3 }}
145
+
run: |
146
+
nix-store --realise "$NIKS3"
147
+
echo "$NIKS3/bin" >> "$GITHUB_PATH"
148
+
- id: build
149
+
env:
150
+
MATRIX: ${{ needs.eval.outputs.matrix }}
151
+
EVAL_MANIFEST_VERSION: ${{ needs.eval.outputs.manifest-version }}
152
+
run: |
153
+
matrix=$RUNNER_TEMP/matrix.json
154
+
plans=$RUNNER_TEMP/plans
155
+
printf '%s' "$MATRIX" > "$matrix"
156
+
read -ra installables <<<"$(jq -r '[.include[].installables] | join(" ")' "$matrix")"
157
+
"$HESTIA_BIN" prefetch "${installables[@]}"
158
+
.github/ci-shared.sh "$matrix" "$plans"
159
+
mapfile -t shared < "$plans/shared.drvs"
160
+
version=$EVAL_MANIFEST_VERSION
161
+
if [ "${#shared[@]}" -gt 0 ]; then
162
+
# a failed one fails the jobs of the machines that need it
163
+
.github/ci-build.sh "${shared[@]/%/^*}" ||
164
+
echo "::warning::some shared derivations failed to build"
165
+
# upload now instead of in the post step, so the machine jobs can
166
+
# wait for the manifest version that contains the shared outputs
167
+
"$HESTIA_BIN" drain --socket "$HESTIA_SOCKET" 2>&1 | tee "$RUNNER_TEMP/drain.txt"
168
+
drained=$(sed -nE 's/.*; manifest m[0-9]+#([0-9]+)$/\1/p' "$RUNNER_TEMP/drain.txt")
169
+
version=${drained:-$version}
170
+
fi
171
+
echo "manifest-version=$version" >> "$GITHUB_OUTPUT"
172
+
{
173
+
for name in $(jq -r '.include[].name' "$matrix"); do
174
+
echo "- $name: $(wc -l < "$plans/$name.drvs") derivations to build"
175
+
done
176
+
echo "- shared (needed by >= 2 machines): ${#shared[@]}"
177
+
printf '%s\n' "${shared[@]}" | sed -E 's|^/nix/store/[^-]+-(.*)\.drv$| \1|'
178
+
} | tee -a "$GITHUB_STEP_SUMMARY"
179
+
180
+
build:
181
+
needs: [eval, shared]
182
+
if: needs.eval.outputs.any-jobs == 'true'
183
+
name: ${{ matrix.name }}
184
+
runs-on: ${{ matrix.os }}
185
+
permissions:
186
+
contents: read
187
+
actions: read # hestia: detect evicted cache entries upfront
188
+
id-token: write # niks3 push
189
+
strategy:
190
+
fail-fast: false
191
+
matrix: ${{ fromJSON(needs.eval.outputs.matrix) }}
192
+
steps:
193
+
- uses: actions/checkout@v4
194
+
- uses: cachix/install-nix-action@v31
195
+
with:
196
+
extra_nix_config: |
197
+
trusted-users = root runner
198
+
extra-substituters = https://cache.lassul.us
199
+
extra-trusted-public-keys = cache.lassul.us-1:aOMEO5TU6eLvhyPi0jmgjR2sVlGjMq3rK0MeACjTcQ0=
200
+
- uses: Mic92/hestia@v3
201
+
with:
202
+
upstream-cache-filter: true
203
+
wait-manifest-version: ${{ needs.shared.outputs.manifest-version }}
204
+
- env:
205
+
NIKS3: ${{ needs.eval.outputs.niks3 }}
206
+
run: |
207
+
nix-store --realise "$NIKS3"
208
+
echo "$NIKS3/bin" >> "$GITHUB_PATH"
209
+
- name: build ${{ matrix.name }}
210
+
env:
211
+
INSTALLABLES: ${{ matrix.installables }}
212
+
run: |
213
+
read -ra installables <<<"$INSTALLABLES"
214
+
"$HESTIA_BIN" prefetch "${installables[@]}"
215
+
.github/ci-build.sh "${installables[@]}"
84
216
85
217
# Fixed-name summary of the jobs above, the required status check for
86
218
# merging into master (the matrix job names change with the hosts).
219
+
# Skipped: no machine to build, or a job it needs failed.
87
220
ci:
88
-
needs: [list, check, outputs, farm]
221
+
needs: [list, check, outputs, eval, shared, build]
89
222
if: always()
90
223
runs-on: ubuntu-latest
91
224
steps:
···
94
227
run: |
95
228
echo "results: $RESULTS"
96
229
for r in $RESULTS; do
97
-
[ "$r" = success ] || exit 1
230
+
[ "$r" = success ] || [ "$r" = skipped ] || exit 1
98
231
done
+2
-2
.github/workflows/update-pkgs.yml
+2
-2
.github/workflows/update-pkgs.yml
···
139
139
140
140
if [ -z "$(gh pr list --head "$branch" --state open --json number --jq '.[].number')" ]; then
141
141
gh pr create --base master --head "$branch" --title "$msg" \
142
-
--body "Automated update by the update-pkgs workflow (\`nix run .#update-pkgs -- $PKG\`), package built on $SYSTEM. Merged automatically once ci (including the farm build of all machines) passed."
142
+
--body "Automated update by the update-pkgs workflow (\`nix run .#update-pkgs -- $PKG\`), package built on $SYSTEM. Merged automatically once ci (including the build of all machines) passed."
143
143
fi
144
144
if [ "$AUTOMERGE" = true ]; then
145
145
gh pr merge "$branch" --auto --merge --delete-branch
···
182
182
fi
183
183
184
184
{
185
-
echo "Automated \`nix flake update\` by the update-pkgs workflow. Merged automatically once ci (including the farm build of all machines) passed."
185
+
echo "Automated \`nix flake update\` by the update-pkgs workflow. Merged automatically once ci (including the build of all machines) passed."
186
186
echo
187
187
echo '```'
188
188
sed -n '/^• /,$p' "$RUNNER_TEMP/update.log"