superconfig/configs/monitoring/ at a047e739ba5a73412c70b4659da969aebfd15536 · lass.pds.lassul.us/superconfig · Tangled

lass.pds.lassul.us / superconfig

0

This repository has no description

0

superconfig / configs / monitoring

1 folder 5 files

at a047e739ba5a73412c70b4659da969aebfd15536

Hermes machines.neoprism: replace pocket-id with zitadel on id.lassul.us 2d ago

configs.monitoring: export the superconfig revision each machine runs autoUpgrade pins the upgrade to the revision the forward-only guard resolved and writes it to /var/lib/superconfig-revision after a successful upgrade, like deploy does. Vector reads that file every 30s and exports nixos_system_info{host,revision,revision_short}; the node dashboard shows host -> revision_short.

5 days ago

configs.monitoring: alert on disk health and memory pressure smartctl_exporter runs on every non-container machine, bound to localhost and scraped by Vector into the host-tagged metrics path. Alerts on a failed SMART status, NVMe critical warnings, new NVMe media errors and NVMe spare below its threshold. Memory PSI comes from /proc/pressure/memory via an exec source, as a counter of stalled seconds, so the alert takes an exact 10m rate instead of the kernel's 5m avg300 EWMA. Plus an alert on any OOM kill. MemoryPressure threshold 10%: some task stalling on memory a tenth of the time over 10 minutes is sustained reclaim/swap thrash, while short spikes such as big builds average out.

4 days ago

configs.monitoring: exempt every mount of the download dataset from DiskAlmostFull yellow mounts the same ZFS dataset (tank/download) as neoprism's /var/download, which is covered by DownloadSpace*, so DiskAlmostFull fired for it. Match the exemption on the device of neoprism's series instead of host and mountpoint.

3 days ago

machines.neoprism: replace pocket-id with zitadel on id.lassul.us Zitadel 2.71 from nixpkgs as a federated IdP with open self-registration: registration, external IdPs and passkeys allowed in the default login policy, first instance "lassul.us" with admin user lass (lass@lassul.us). The masterkey and the admin's initial password come from the zitadel clan vars generator, the password via an extra steps file so it never hits the store. Postgres via peer auth over the local socket. ensureDatabases/ensureUsers create the database and owner role, so the zitadel role gets no CREATEROLE/CREATEDB (on PG15 CREATEROLE is close to superuser). That needs `init zitadel` + start-from-setup instead of the module's start-from-init, whose init always tries CREATE USER/CREATE DATABASE and fails on the permission check before the "already exists" check. nginx terminates TLS and talks h2c to Zitadel via grpc_pass, as Zitadel's reverse-proxy docs require for its gRPC APIs. Machine identification by hostname, since the default private-IP/GCP-webhook probes don't fit neoprism. The database is pg_dumped by the borg preHook; gatus checks /debug/healthz. Zitadel 2.71 does not work with PostgreSQL 18 (migration 34_add_cache_schema creates unlogged partitioned tables), neoprism runs 15.

2 days ago

machines.orange: remove, move mumble-reminder to neoprism

5 days ago

retiolum: drop self.retiolum, acheron and starkstrom are in kartei now kartei carries both cards (krebs/kartei#92), so the superconfig-only host set is empty; read hosts, own addresses and tinc host files from kartei alone.

5 days ago