Introduction

Garret is a single-tenant Nix binary cache. An authenticated Pusher accepts store paths; a public Puller serves them to any machine whose nix.conf points at it. Both run on one host over SQLite and an S3-compatible bucket.

Components

Topology

The two services are separate because they face different audiences: the Pusher decides who may write, and the Puller serves reads to the world. They share one database file and one bucket, so they run on the same host.

Limits

• One tenant and one cache per deployment.
• Both services run on one host and share one database.
• An interrupted upload starts over; there is no resume.

Read How it works for the shape of a push and a pull, or go straight to Installation.

Installation

Garret is not a hosted service. You run the Pusher and the Puller on your own host, and point your machines at the Puller. This page installs the client; Deployment covers the server.

Install the client

nix profile install github:jeiang/garret#garret

Pin a revision with github:jeiang/garret/<rev>#garret to keep a fixed version.

Flake outputs

The NixOS modules services.garret-pusher, services.garret-puller and services.garret-watcher default to these packages.

Point nix at your cache

Once your Puller is running, add it to nix.conf:

extra-substituters = https://cache.example
extra-trusted-public-keys = cache.example-1:<base64 public key>

garret use writes both entries for you. garret-admin key show prints the public key of the signing key your Pusher uses.

On NixOS, set the same values through nix.settings. On a multi-user install, --option on the command line works only for trusted users.

Next: Quickstart to push your first path.

Quickstart

Four commands take you from nothing to a cache your machines substitute from.

garret login https://push.cache.example   # writes the config, then signs you in
garret use                                # points nix at the cache
garret push ./result                      # uploads the closure
garret watch-store                        # keeps pushing what you build

garret login takes the Pusher URL and fetches the rest — Puller URL, signing keys, OIDC issuer and client id — from the server. It creates the config directory if it is absent.

Commands without a config

Exactly these run without one: garret login <url>, garret logout, garret completions <shell> and garret enqueue. garret login without a URL needs an existing config, since there is nothing to re-authenticate against.

Machines without a human

Build hosts and CI skip login. Set [oidc] client_secret_file to a file holding only the client secret, with client_id and issuer alongside it, or set GARRET_TOKEN. A GitHub Actions runner with id-token: write needs neither. GARRET_ENDPOINT overrides endpoint.

Pointing nix at the cache

garret use appends the cache to the user’s nix.conf. Substituters in a user’s nix.conf are ignored unless that user is in trusted-users, so the command checks nix config show substituters afterwards and exits 1 when nix ignores the file, printing the nix.settings snippet to use instead.

The full reference — every command, config key and default — is in the Client reference.

Client reference

garret is the client. It pushes store paths, watches the local Nix store, lists what the cache holds, and diagnoses a broken setup.

Commands

--config <path> is global and overrides the config file for every command that loads one.

list, tree and pins read the Puller’s browse API, which requires sign-in and returns 404 on every path when browse_oidc is unset on the Puller.

Configuration

The config lives at $XDG_CONFIG_HOME/garret/config.toml, falling back to ~/.config/garret/config.toml when XDG_CONFIG_HOME is unset. garret use edits $XDG_CONFIG_HOME/nix/nix.conf with the same fallback.

The [oidc] table and its audience key are required. Only issuer and client_id are optional.

garret login writes endpoint, puller_endpoint, public_keys and the three advertised [oidc] keys — issuer, client_id, audience. It does not write the whole section, and it does not write client_secret_file or resource.

login refuses a discovery document whose URLs are not https (plain http only to a loopback host), and whose values contain characters that would break the file: URLs reject any non-graphic character plus ", \, {, } and backtick; the audience and client id reject any non-graphic character plus " and \. Every command re-checks the URLs and keys when it loads the config, because the file may predate the check or be hand-edited.

Authentication

Every command that talks to a server takes its token from the first source that applies:

1. GARRET_TOKEN, used as given and never renewed.
2. [oidc] client_secret_file, a file holding only the client secret. The client id comes from oidc.client_id; both it and oidc.issuer are required alongside. The client-credentials grant is re-minted on expiry. A warning goes to stderr when the file is readable by group or others.
3. A GitHub Actions runner with both ACTIONS_ID_TOKEN_REQUEST_URL and ACTIONS_ID_TOKEN_REQUEST_TOKEN set.
4. The refresh token stored by garret login.

The token file is 0600 in a 0700 directory and is replaced whole, so no reader sees it half-written. The token provider times out after 60 s, and a token is re-minted 60 s before expiry. Concurrent requests share one mint, which rotating refresh tokens require.

A non-2xx answer from the issuer or the cache is reported with the response body, not just the status. A refresh refused with 400 or 401 reads “stored login expired or revoked; run garret login”. During the device flow, slow_down adds 5 s to the poll interval.

Push behaviour

Every request carries User-Agent: garret/<version>. The Pusher URL, the Puller URL, the issuer, the token endpoint and the device-authorization endpoint must follow the same https rule as discovery, and every configured key must be name:base64; a config that breaks either is refused before any token is sent. Trailing slashes on endpoint and puller_endpoint are dropped on load.

Retries run on two schedules:

• A 429 is the server’s queue, not a fault. The path waits out Retry-After, held between 1 s and 15 minutes plus up to as much again in jitter, and tries again for up to 15 minutes from its first attempt. The deadline is per path, so a run fails after roughly 15 minutes per jobs paths. Sheds never spend max_retries.
• An in-progress answer means another pusher holds the path. The client waits it out against the same deadline, polling after jittered waits that double from 1 s to 30 s, and does not count the path until the server answers “exists” or takes this upload.
• 5xx answers and dropped connections get max_retries jittered waits doubling from 250 ms. A shed or an in-progress wait resets that count, so it bounds consecutive faults.
• Other 4xx answers fail at once. A 403 is a permanent policy refusal: it is never renewed or retried within the push. A 401 renews the token once and retries immediately; a second 401 is an ordinary failure.

Uploads start in references-first order. That is an order of starts, not of completions: with jobs above 1 a referrer can finish before its references, and a failed reference does not hold its referrers back.

After a run with no failures that uploaded something, garret push negotiates the whole closure once more, because a path found present at the start can be gone by the end. Whatever is missing is reported and pushed, and its results are folded into the same summary. There is one re-check, not a loop. If the re-check itself fails, done is still reported and the run exits non-zero.

A failure does not abort the run: every path is attempted and reported, the summary is emitted, and only then does the process exit non-zero.

--dry-run negotiates, reports what would be uploaded, and stops.

zstd_level has no upper bound in the client. A level the server cannot accept is rejected with 400 at upload time.

Progress is a single bar over uncompressed NAR bytes, drawn to stderr and hidden when stderr is not a terminal. There is no --no-progress.

A path reported deduped was claimed by another writer first. Bandwidth may or may not have been spent, because the server answers “already present” before reading the body.

JSON output

--json is accepted globally but honoured only by list, tree, pins, whoami, doctor and push. login, logout and use ignore it and print human text to stdout.

list and tree emit the browse API’s JSON verbatim. whoami and doctor each emit one object. push emits NDJSON, one event per line:

{"event":"negotiated","closure":312,"upstream":118,"missing":47,"nar_bytes":4021374976}
{"event":"path","path":"/nix/store/…","status":"upstream","nar_size":123456}
{"event":"path","path":"/nix/store/…","status":"pushed","nar_size":81920}
{"event":"path","path":"/nix/store/…","status":"deduped","nar_size":4096}
{"event":"path","path":"/nix/store/…","status":"failed","error":"upload rejected with 503: …"}
{"event":"done","pushed":40,"deduped":6,"failed":1,"nar_bytes":4021374976}

A run with no failures may end with a re-check before done:

{"event":"rechecked","missing":1,"nar_bytes":81920}
{"event":"path","path":"/nix/store/…","status":"pushed","nar_size":81920}
{"event":"done","pushed":48,"deduped":0,"failed":0,"nar_bytes":4021456896}

negotiated gives a consumer the denominator up front, and done makes a truncated stream detectable. --dry-run --json reuses the same schema, with status:"would-push" for what would upload and a done whose pushed, deduped and failed are zero while nar_bytes is the total that would have been uploaded.

Store watcher

garret watch-store pushes store paths as they are built.

• Cursor. A persisted position in the local Nix database’s history of validated paths. Catch-up after downtime, the initial scan and an offline backlog are all “the cursor is old”. When no cursor file exists it starts at the newest path, so only new paths push; --full-sync walks history from the beginning instead.
• Wake socket. A unix datagram socket at [watch] socket_path, mode 0666. Any datagram means “poll now”, and bursts coalesce into one early poll. garret enqueue is the sender, registered as nix’s post-build-hook, so pushes start the moment a build finishes. A socket left by an earlier run is replaced; anything else at the path is refused rather than deleted. A bind failure only warns, and enqueue exits 0 even when nothing listens: a hook must never block or fail a build.
• Batching. Each poll, and each drain chunk of 500, is one batch: the paths that pass the filters go through a single non-recursive nix path-info, then a single negotiation naming the batch plus its direct references, and only the batch’s own missing paths upload, jobs concurrent, references first.
• Filtering. Skips .drv paths, exclude_patterns, and paths already signed by a configured upstream_keys key or by one of the cache’s own public_keys. Signatures are read from the local Nix database. -source and fixed-output paths are kept.
• Failures. The cursor always advances, so one bad path never wedges the pipeline. A path whose push fails goes on the failed list at <cursor_path>.failed, one store path per line, and the cursor is written once per batch after the failures are on the list. The daemon retries the list whenever a poll finds nothing new, up to max_attempts per path; the failure that spends the last attempt logs loudly and leaves the path for a restart or a drain. Paths removed by nix GC before the push are dropped silently. The cursor and the list are each replaced whole by a rename, so a kill mid-write cannot lose the backlog. A cursor or list that exists but cannot be read stops the watcher with an error naming it.
• Drain. --drain pushes every path validated after the cursor plus the failed list, in batches of 500, then exits — non-zero, naming the paths, if any failed. Run it after stopping the daemon, which shares the cursor. With no cursor it refuses rather than guessing where to start, unless --full-sync.

Hook setup

post-build-hook takes a bare program with no arguments, so it cannot name garret enqueue directly. The NixOS module wires a wrapper automatically whenever the watcher is enabled. Elsewhere, write one:

#!/bin/sh
exec garret enqueue

and set post-build-hook = /path/to/that-script in nix.conf.

Push as built in CI

A job that fails late should still leave behind everything it built, so the next run substitutes it. The shape: the hook wakes a background watch-store as each build finishes, and an always-run last step stops it and drains. No secret is needed: on a GitHub Actions runner with id-token: write, garret mints its push tokens from the runner, and oidc.issuer is not required.

permissions:
  contents: read
  id-token: write # garret mints push tokens from the runner

steps:
  - uses: actions/checkout@v5

  # Before nix is installed: the hook must exist before the first build,
  # and must succeed while garret itself is not installed yet.
  - name: Write garret's post-build hook
    run: |
      cat > "$RUNNER_TEMP/garret-hook" <<EOF
      #!/bin/sh
      [ -x "$RUNNER_TEMP/garret/bin/garret" ] || exit 0
      exec "$RUNNER_TEMP/garret/bin/garret" enqueue --socket "$RUNNER_TEMP/garret.sock"
      EOF
      chmod +x "$RUNNER_TEMP/garret-hook"

  - uses: cachix/install-nix-action@v31
    with:
      extra_nix_config: |
        experimental-features = nix-command flakes
        post-build-hook = ${{ runner.temp }}/garret-hook

  - name: Start pushing as paths are built
    run: |
      nix build --out-link "$RUNNER_TEMP/garret" github:jeiang/garret#garret
      cat > "$RUNNER_TEMP/garret.toml" <<EOF
      endpoint = "https://push.cache.example"

      [oidc]
      issuer = "https://token.actions.githubusercontent.com"
      audience = "garret"

      [watch]
      cursor_path = "$RUNNER_TEMP/garret-cursor"
      socket_path = "$RUNNER_TEMP/garret.sock"
      EOF
      nohup "$RUNNER_TEMP/garret/bin/garret" --config "$RUNNER_TEMP/garret.toml" \
        watch-store > "$RUNNER_TEMP/garret-watch.log" 2>&1 &
      echo $! > "$RUNNER_TEMP/garret-watch.pid"
      # The cursor appears once it has authenticated and bootstrapped.
      for _ in $(seq 30); do [ -s "$RUNNER_TEMP/garret-cursor" ] && exit 0; sleep 1; done
      cat "$RUNNER_TEMP/garret-watch.log"; exit 1

  - run: nix build .#everything

  - name: Push whatever the watcher has not
    if: always()
    run: |
      kill "$(cat "$RUNNER_TEMP/garret-watch.pid")" || true
      "$RUNNER_TEMP/garret/bin/garret" --config "$RUNNER_TEMP/garret.toml" watch-store --drain

Doctor

garret doctor [path] runs its checks in layer order, prints one pass, fail or skip line per check, and exits non-zero if any check failed.

A check whose prerequisite failed reports skip rather than guessing. The keys check runs for every device-flow config and for any config with a non-empty public_keys; only a machine config with no keys skips it. Its messages suggest garret login --force even for a machine config — only the config drift hint is device-flow-aware.

Timeouts: the discovery and pull probes are capped at 15 s, the auth check may spend up to 60 s minting a token, and the two negotiations use 2 minutes.

Under --json, doctor emits one object; the failure summary goes to stderr:

{"ok":false,"checks":[{"name":"discovery","status":"pass","detail":"…"},
                      {"name":"path","status":"fail","detail":"not cached — …"}]}

Deployment

Garret runs as two systemd services on one NixOS host, plus an optional store watcher on build machines. The NixOS modules generate the service config from typed options; a hand-written config works too, and the binaries check the same ranges at startup.

Modules

The Puller has no database path, store directory or bucket options of its own: it serves the Pusher’s database and bucket on the same host, so it takes them from services.garret.pusher. Enabling the Puller without the Pusher fails evaluation.

Pusher options

An option marked null is left out of the generated config, so the binary applies its own default.

pullerEndpoint is the public URL of the Puller. The Pusher advertises it, with the signing keys and the OIDC settings, through /api/v1/discovery, which is how garret login <pusher-url> writes a whole client config.

signingKeyFiles takes several keys during a rotation: every object is signed with all of them, and garret-admin resign backfills the ones added later.

Issuer submodules

Each entry in oidc takes issuer, audience, extra_audiences, client_id, jwks_url, github_owner_id, ref_patterns, ref_protected, repository_ids, event_names, job_workflow_refs and allowed_groups.

Puller options

browseOidc takes issuer, audience, jwks_url and allowed_groups. When it is unset the browse API is not served at all: every browse path returns 404. Any configured issuer is accepted there; it need not be Pocket ID.

Give the Puller its own S3 key limited to GetObject. A presigned URL carries exactly the authority of the key that signed it, so the Pusher’s key would hand write access to the public-facing process.

Watcher options

Enabling the watcher also sets nix.settings.post-build-hook to a wrapper that runs garret enqueue, so a build wakes the watcher the moment it finishes.

Authentication

The Pusher validates bearer tokens from the issuers you configure, directly against each issuer’s published keys. There is no token-exchange service and no garret-issued token, and no way to disable authentication.

• Pocket ID for interactive use. Register a garret API and audience there, and restrict the client to the right user group in Pocket ID. An optional allowed_groups list adds a second check.
• GitHub Actions for CI. Match the immutable repository_owner_id claim, never a renameable name. Owner-wide matching alone lets every workflow in every repository of that owner mint a push token, so narrow it with repository_ids, ref_patterns, ref_protected, event_names and job_workflow_refs. Each is off when unset; once set, a token missing the claim is refused. The Pusher refuses to start when a GitHub issuer sets neither github_owner_id nor repository_ids.

A token that fails verification gets 401, and the client may renew or retry. A token that verifies but that a policy gate refuses gets 403, which is permanent until the policy changes; clients never renew on it.

Refresh-token lifetime is entirely the issuer’s business; garret neither sets nor checks it.

Service users and file modes

The Pusher and the Puller run as different users, so a compromise of the public-facing Puller reaches neither the signing keys, the admin socket, nor bucket write.

The Puller is read-only except for debounced last-accessed bumps, which is why it needs write access to the database files.

Signing keys must not be readable by group or others. The loader accepts any mode without group or other bits — 0400 and 0600 both load — and refuses anything else, naming the file and its mode.

The database directory and files are created with the modes above by the Pusher’s start step. A database put in place by hand must be owned by garret.

Process behaviour

Both server units are Type=notify: the binaries send READY=1 once every listener is bound, so units ordered after them start only when the sockets accept connections.

All three units share one sandbox: an empty capability bounding set, NoNewPrivileges, ProtectSystem=strict with only the unit’s own state writable, ProtectHome, private /tmp and /dev, MemoryDenyWriteExecute, and a system-call filter. The watcher runs as root with no capabilities, because everything it touches is root-owned or world-accessible.

Size the Pusher’s MemoryMax at no less than 2 × maxInFlightBytes + maxConcurrentUploads × 8 MiB + ~1 MiB × maxConcurrentUploads, plus headroom for the process. The defaults need about 700 MiB.

Scheduled fsck

fsck.calendar takes a systemd OnCalendar expression and adds garret-fsck.service and garret-fsck.timer. It requires quotaBytes, because fsck reads the orphan grace from the [gc] section.

The unit runs garret-admin fsck with fsck.extraArgs appended verbatim. The default is a read-only dry run, but the list is unconstrained: extraArgs = ["--repair"] makes the scheduled unit delete rows, and ["--repair", "--quiesce"] also rejects pushes while it runs. Set them deliberately.

The unit exits 1, and so shows as failed, when a run finds dangling rows or size mismatches. Orphan blobs alone do not fail it.

Validation

The module types reject most of what the binaries reject, at evaluation time: positive integers for the quota, GC interval, S3 timeouts and limits, watermarks between 0 and 1, and so on. They are not identical. Watermarks accept 0.0 and low == high, and operationAttemptTimeoutSeconds may be greater than or equal to operationTimeoutSeconds; the Pusher refuses to start in both cases. A hand-written config gets the same startup checks.

Raw settings

extraSettings is raw TOML merged recursively over the generated config, for keys without a typed option. For example, extra audiences and a client id on the browse API:

services.garret.puller.extraSettings.browse_oidc = {
  extra_audiences = [ "garret-browse" ];
  client_id = "garret-browse";
};

Secrets — S3 credentials, signing keys, OIDC client secrets — are file paths, never values in the nix store.

Operations

Day-to-day work on a running deployment: keeping the cache inside its budget, removing what should not be there, backing it up, and watching it.

Quota and garbage collection

Garbage collection is optional. The [gc] section is what turns it on: without it there is no quota, no eviction and no orphan sweep, the Pusher logs that the cache is unbounded and will never evict, and both garret-admin gc run and garret-admin fsck fail with no [gc] section is configured.

Eviction starts when usage reaches the high watermark — 95% of the quota by default — and stops at the low watermark. Below the high watermark nothing is deleted.

An object is removed only when no other object in the cache references it, no live pin holds it, and no push used it within the 24-hour push grace. Candidates go in last-accessed order, so old roots go first and deleting them frees their dependencies for the same pass. If everything left is referenced, pinned or inside the grace, the pass stops and raises an alarm metric rather than breaking a closure.

The push grace exists because a client that was told a path is present will not upload it, and will upload its referrers afterwards. Until those land, the path looks like an unreferenced root.

Pins

A pin is a GC-exempt root that protects its whole closure.

garret-admin pin <name> <hash> [--expires <duration>]
garret-admin unpin <name>

Pinning a hash the cache does not hold is a hard error. An expired pin simply stops matching; nothing sweeps it. Pinned bytes still count against the quota, so a quota made entirely of live pins shows up as the alarm above rather than silent thrashing.

Prune

garret-admin prune --before <YYYY-MM-DD|age> [--apply]

Deletes every closure last pushed before the cutoff, regardless of quota. It keeps everything pushed at or after the cutoff, every live pin, and each one’s whole reference closure. The cutoff must be at least 24 hours in the past.

Age is the time of the last push, not the time the object was first stored: CI re-pushing an unchanged closure uploads nothing, but its negotiation marks every path it names as pushed now. Without that, a system closure unchanged for months would look old while still in daily use.

Without --apply it lists what it would delete and takes no write lock.

Delete

garret-admin delete <hash>... [--apply]
garret-admin delete --pushed-by <subject> [--since <YYYY-MM-DD|age>] [--apply]

Removes objects unconditionally, row then blob, whatever still references them. Without --apply it lists what would go.

--pushed-by is the incident-response form: it takes a subject as the browse API shows it (<issuer>#<sub>) and selects every object that subject first pushed at or after the cutoff.

Wherever a command takes a <hash>, it also accepts the store path (/nix/store/<hash>-<name> or <hash>-<name>). Any other argument is rejected before the request is sent, so a typo fails loudly instead of coming back as “not in the cache”.

fsck

garret-admin fsck [--repair] [--verify-sizes] [--quiesce] [--json]

Audits the database against the bucket. It reports dangling rows (a row with no blob) and orphan blobs (a blob with no row); --verify-sizes adds a third category, rows whose recorded size disagrees with the stored object.

The default is a dry run. --repair deletes dangling and size-mismatched rows; it never deletes an S3 object, so a size-mismatched row’s blob is left for the orphan sweep. Only the row the listing saw is deleted: a row that changed since is skipped and not counted as repaired.

--quiesce shuts the upload gate, answers new uploads with 503 and Retry-After: 1, and waits for in-flight uploads to drain before repairing. The drain is bounded at a fixed 300 seconds — there is no flag or config key for it — and a timed-out drain repairs nothing and reports quiesce_drained: false.

--json emits the report plus ok, and exits 1 when the run is not ok.

Status

garret-admin status

Prints the object count, usage against the quota, and the number of in-flight uploads.

Backup and restore

garret-admin backup <path>

Takes an online copy of the database while both services keep serving. The copy is a consistent snapshot including the uncheckpointed write-ahead log, is created mode 0600, and never overwrites an existing path. Rotation is your job: remove or rename the previous copy first.

The path is resolved by garret-admin and written by the Pusher, so it must be writable by the Pusher and visible outside it. Under the NixOS module, put it in the directory holding dbPath.

To restore a copy that is older than the bucket:

1. Stop both services. Put the copy in place of dbPath, owned by garret, and delete any -wal and -shm files beside it.
2. Start only the Pusher, with its push endpoint unreachable to clients.
3. garret-admin fsck --repair --verify-sizes --quiesce deletes the rows whose blobs were evicted after the backup, and rows whose blob a later delete and re-push replaced with one of a different size.
4. Start the Puller.
5. Once every restored row is older than orphan_grace_secs, run step 3 again, then reopen the push endpoint.
6. Re-apply any pins set or removed since the backup.

Closures stay complete because eviction and prune delete an object only once nothing surviving references it, roots first. Anything deleted after the backup was therefore unreferenced by every object still present. Blobs uploaded after the backup have no row; clients re-push them through normal negotiation, and the orphan sweep removes the rest. That costs cache misses, never correctness.

The ordering matters: negotiation answers from rows alone, so while a restored row is unchecked it can tell a client that a path whose blob is gone is present. Keeping pushes closed until fsck has covered every row is what prevents that.

Last-accessed bumps

The Puller records last-accessed times for eviction ordering, but only when the stored value is stale. The threshold is bump_debounce_secs, default 86400. A burst of hits on one object is one write, flushed in a batch every few seconds.

Monitoring

The Pusher exposes metrics on port 9091 and the Puller on 9092, each with /metrics, /healthz, /ready and /ready/deep. The public listener never exposes metrics.

• /ready answers 200 once the database is open and 503 before.
• /ready/deep also performs one real read through the path a substituter follows: it presigns the most recently accessed object’s blob and fetches its first byte. Probe this one rather than /ready, at 30-second intervals or slower, and alert on a few consecutive failures. The answer is cached for 30 seconds, so it costs at most one small read per half minute.

Alert rules and a Grafana dashboard ship in contrib/monitoring/. The rules assume each service is scraped under a job name starting with garret; if yours differ, edit the GarretDown selector.

The Puller makes no S3 API calls. Its only S3 traffic is the /ready/deep read of a presigned URL, which is why revoked or rotated credentials fail that probe while narinfo and NAR redirects keep working.

How it works

Garret is two services over one database and one bucket. This page is the shape of the system, not its wire formats.

Why two services

The split is about exposure and authentication, not placement. The Pusher decides who may write and is never public; the Puller serves reads to anyone and holds no authority to write. Both run on one host and share one database file and one bucket, so there is no network hop between them and no second copy of the metadata.

Push

The client sends the whole closure once to ask what is missing, then uploads only what the cache lacks. Each path is one compressed NAR, streamed rather than buffered.

The Pusher checks the store path and the claimed hash before it reads the body, and answers “already present” immediately if another writer got there first. The client may still be sending at that point, so a deduplicated path may still have cost bandwidth.

The object’s row is written after its blob. That ordering is the invariant: a row never points at a missing blob. An orphaned blob with no row is possible and expected, and a periodic sweep removes it.

Pushes are idempotent. Two clients pushing the same closure, or a client retrying after a dropped connection, produce the same result.

Pull

The Puller signs narinfos with the cache’s ed25519 keys, so nix verifies them like any other cache. It never proxies NAR bytes: it answers with a redirect to a URL that grants one read for a limited time, and the client fetches the bytes from the bucket directly.

That keeps the Puller small and fast, and it means the bucket must be reachable by the machines that substitute. It also means a pull that cannot reach the bucket fails at the client, not at the Puller.

Storage

One compressed NAR per store path, keyed by the path’s hash. There is no chunking, no content deduplication and no reference counting. Reuse happens at path granularity: the negotiation step is what stops a client uploading something the cache already has.

The cost of that choice is that two store paths with nearly identical contents are stored twice. The benefit is that every object is one blob, so reads, deletes and repairs are single operations.

Metadata

One SQLite database in WAL mode, shared by both services. The Pusher performs every write: objects, references, garbage collection, signatures. The Puller is read-only except for one thing — it records last-accessed times, in a debounced batch, because eviction orders by them.

Because the Pusher is the only writer, there is no coordination protocol between the services: no locks, no leader election, no queue.

Signing

The cache holds one or more ed25519 signing keys. Narinfos are signed with all of them, so a key rotation can overlap: clients that trust either key keep working. garret-admin resign backfills signatures after a key is added, and refuses to sign a row whose existing signatures do not verify under a configured key, so tampered metadata never gains a valid signature.

Garbage collection

A quota with a high and a low watermark. Eviction removes the least-recently-accessed objects first, and only objects that nothing else needs: no surviving object references them, no pin holds them, and no push used them within the push grace. Pins exempt a whole closure. Prune removes everything last pushed before a cutoff, independent of the quota, with the same safety rules.

The result is that every closure the cache still serves stays complete. A cache can run out of evictable objects before it reaches its low watermark; when that happens it stops and raises an alarm rather than deleting something a closure needs.

When something is wrong

• A pull that cannot meet its time budget answers 404 and increments a counter rather than hanging. A cache miss and a degraded lookup look the same to nix, which then builds or substitutes elsewhere.
• A database that is not ready yet answers 503, so the Puller can start before the Pusher has created the file.
• An upload that arrives while its object is being deleted, or while a repair is draining uploads, gets 503 with a retry hint. The client waits it out on its own deadline instead of spending retries.

Benchmarks

Measured on garret itself, against a throwaway local Garage standing in for the object store. The numbers describe the cache, not the bucket.

What is measured

• Push: 20 concurrent pushers over a seeded corpus of 200 entries totalling 638 MiB, each entry 4 KiB to 16 MiB.
• Stream: single-stream pushes of 1 MiB, 100 MiB and 2 GiB bodies.
• Pull: 1200 concurrent narinfo and redirect requests.

The corpus is generated from a seed, so a run is reproducible without holding real store contents.

Systems

The Ryzen server limits apply to the garret services only: taskset -c 0 for the CPU pin, and per-service systemd user scopes with MemoryMax=896M MemorySwapMax=0 for the memory cap. Garage and the bench client stand in for out-of-sandbox components — the upstream S3 service and the pushing machines — and run unconstrained.

Results

The Pusher runs the default limits — a 256 MiB in-flight cap, 16 MiB parts, 4 parts in flight — except max_concurrent_uploads, raised from the default 16 to 32 so the 20 pushers are not shed.

Pass criteria

A run passes when it has zero failed requests and zero shed retries. The memory criterion is that peak Pusher RSS stays below twice the 256 MiB in-flight cap, that is 512 MiB. It held in every configuration above.

just bench-local enforces the peak. Final RSS is recorded in the JSON for comparison and is not gated.

Run-to-run variance

A second pass on the Mac agreed within 2% on push, the 1 MiB and 100 MiB streams, pull and RSS; the 2 GiB stream came in 5% higher and push p99 8% higher. On the Ryzen server, a second pass of the matrix agreed within 9% on push, streams and RSS, but not on pull: the 1200-request scenario lasts about 30 ms on 8 cores, and the two unpinned configurations came in 26–29% lower on the second pass, while the 1-CPU configurations agreed within 1%.

Treat the unpinned Ryzen pull column as indicative only. For pull comparisons, run GARRET_BENCH_ARGS="--passes 100 --no-giant" just bench-local for 40,000 requests. Most such runs agree within about 2%, but about one in five came in 4–18% low, so compare the best of three.

Why max is reported

Roughly half of the 1-CPU push runs catch a Garage artifact: a single push held open for seconds — occasionally forever, with megabytes unread in Garage’s socket receive queue — while every percentile stays normal, because with 200 entries p99 is only the second-worst sample. It reproduces with Garage unpinned on free cores and on tmpfs, with zero TCP retransmits and zero client retries, so it is a stand-in artifact rather than garret behaviour.

max_ms is the tell. A run whose max_ms is out of line with p99_ms, or whose wall time is far above the sum of latencies divided by concurrency, caught the stall and should be re-run rather than checked in.

Reproducing

just bench-local provisions a throwaway Garage and runs every scenario against it. just bench-compare diffs a fresh run against baseline.json and refuses to compare across labels or corpus versions; each environment keeps its own file. The checked-in result JSONs live in benchmarks/.

Agent guide

Garret is a single-tenant Nix binary cache: an authenticated Pusher that accepts store paths, and a public Puller that serves them. It is a Rust workspace of separate service binaries over SQLite and an S3-compatible bucket.

Where to look

Commands

Run from the repository root, inside the Nix development shell.

just check and just test both pass before work is called finished.

Reading this documentation as an agent

• https://garret.docs.jeiang.dev/llms.txt — one line per page, with a description.
• https://garret.docs.jeiang.dev/llms-full.txt — the whole book in one Markdown file.

The repository’s AGENTS.md carries the working agreements and the rules for changing the code.

Invariants

When reasoning about behaviour, these hold:

• The Pusher performs every metadata write. The Puller only records last-accessed times, in a debounced batch.
• An object’s row implies its blob. The reverse is not true: an orphaned blob is an expected state that the orphan sweep removes.
• Eviction and prune never remove an object that a surviving object, a newer push, or a live pin still needs.
• A push is idempotent. Two clients pushing the same closure, or a client retrying, produce the same result.

Glossary

Garret

This project: a single-tenant Nix binary cache with an authenticated Pusher and a public Puller.

Pusher

The service that accepts NARs from clients over a purpose-built protocol. Protected by OIDC (Pocket ID and GitHub Actions issuers). Not exposed publicly.

Puller

The public binary cache frontend. Speaks the standard Nix substituter protocol (narinfo and NAR over HTTP, ed25519-signed narinfos) so any machine’s nix.conf can use it without special tooling. Read-only except for debounced last-accessed bumps, which eviction ordering needs.

Garret Client

The CLI that drives the Pusher: pushing store paths, watching the local Nix store, and listing cache contents. Pull never requires it.

Store Watcher

The client mode that observes the local Nix store and pushes newly built store paths to the Pusher automatically.

Object

The unit of content in the cache: one Nix store path mapped into the cache, keyed by its store path hash.

Blob

The stored form of an object: a single compressed NAR. Exactly one blob per object. The invariant is one-way — an object’s row implies its blob, never the reverse. An orphaned blob with no row is an expected state that the orphan sweep removes.

Negotiation

The pre-upload exchange in which a client asks which of a batch of store-path hashes the cache is missing, and pushes only those.

Watcher Cursor

The store watcher’s persisted position in the local Nix store’s history of validated paths. Everything after the cursor is yet to be pushed; an old cursor is a backlog, not an error. A path the cursor passed whose push failed is kept on the failed list beside it until a retry, restart or drain pushes it.

Quota

The configured storage budget for the cache. Eviction reclaims space when usage reaches the high watermark, and deletes nothing below it.

Eviction

Removing an object and its blob to reclaim quota. It starts at the high watermark (default 95% of the quota) and stops at the low watermark (85%). Only objects no surviving object references, no live pin holds, and no push used within the push grace may be evicted, least-recently-accessed first — so every closure the cache still serves, or a push is still relying on, remains complete.

Push Grace

The day after a push or negotiation last used an object, during which neither eviction nor prune removes it: the client was told the object is present and uploads its referrers against that answer.

Prune

Operator-requested removal of every closure last pushed before a cutoff, independent of quota. Like eviction, it never removes an object that a surviving object, a newer push, or a live pin still needs.

Pin

A named GC-exempt root. Pinning protects the root’s whole closure, because eviction is root-first. Pinning a hash the cache does not hold is an error, and an expired pin simply stops matching.

Orphan Sweep

A periodic pass, at startup and weekly, that deletes blobs with no database row and aborts abandoned multipart uploads. It is what makes the one-way blob invariant safe: rows are deleted before blobs, so a failed blob delete leaves an orphan for the sweep rather than a row pointing at nothing.

Deletion Claim

The Pusher’s in-memory hold on an object being removed — by eviction, prune, or an operator delete — from before its row is deleted until its blob is. An upload of that object meanwhile is told to retry, so it cannot write a blob the pending delete would take.

Discovery

The anonymous endpoint the Pusher serves so a client can configure itself: the Puller URL, the signing keys’ public halves, and the OIDC issuer, audience and client id. It carries nothing secret and reveals no cache contents.

Upstream Filter

The client-side rule that skips store paths an upstream cache already serves signed, so pushing them would spend bandwidth and quota on bytes the cache never needed to hold. It is a signature-name comparison, with no network probe.

Upstream Keys

The key names the upstream filter recognises. Defaults to ["cache.nixos.org-1"]; the watcher can override the list per daemon.