Skip to content

Storage: make the retention tool the general cleaner (scratch, worktrees, docker, dedupe, compress-cold, build-cache) #470

Closed

Description

@scuttletoby

A storage study on the shared agent machine (2026-11-18) found about 124 GB that can be freed without losing data we need. The existing retention tool misses all of it by design:

  • It skips agent session scratch folders.
  • It keeps every scratch entry that a tracked file mentions.
  • It runs its Docker rule on one host only.
  • Its worktree pruner knows only one repo.

Plan: do not write a new tool. Make the existing retention tool (plan / run, in-use checks, KEEP files, a log line per delete, config, tests) the general cleaner. The rules live in the public fork tooling; host and project facts go in a config file per host.

Rules to add:

Rule What it does
agent-scratch Session scratch idle 3+ days. Needs the owner's yes (separate issue).
worktrees For any repo listed in config. Same refusals as the current pruner: open PR, no merged PR, merged under 24 h ago, dirty or untracked files, commits not in the PR, a process inside. A full clone goes only after its worktrees.
docker On every host listed: anonymous (64-hex) volumes unused by any container and 7+ days old. Named volumes never.
dedupe Identical files of 1 MB or more, untouched for 7+ days, in listed scratch roots become hardlinks. Each is linked under a temp name and then renamed, so a file is never missing.
compress-cold Listed spool folders are packed into .tar.zst. Originals go only after zstd -t and a file count check pass.
build-cache Listed folders, with their rebuild command.
tmp Patterns get a 1-day life (crash-test-, fork-ci-release-test, tmp.*).

Refusals in code, not only in config:

  • Must-keep classes: agent transcripts, Codex sessions, T3 databases and logs, the log archive, and repair backups. Only compress-cold may touch these, and only per path after the owner turns it on.
  • Anything in use: a process's working directory, an open file, a path a systemd unit names, a lock file.
  • Git folders with unpushed commits or uncommitted changes, and anything with an open PR.
  • Anything behind a symlink, on another filesystem, owned by another user, or looking like a credential.
  • Any case where GitHub or the process list could not be read: no answer means keep.

Report: one daily line with GB per class and the change since yesterday.

General vs project: the rules and refusals are general (any host, any repo). Repo lists, roots and spool folders come from each project's config.

State question: each rule has to say what happens when it runs again, stops half-way, or meets a file in use. The study notes give the answer for each one.

🤖 Generated with Claude Code

Activity

  1. scuttletoby commented on Nov 21, 2026

    @scuttletoby
    OwnerAuthor

    Done in fork-kit/cleaner (#492, #499, #500, #511, #512, #524, #563, #564, #603, #620, #635): worktrees, docker, dedupe, tmp, compress-cold, agent-scratch, every refusal in code, the journal for half-done deletes, and the daily report. build-cache is not planned (#467). Two small differences from the wording, left on purpose: the report shows what runs freed since the last report rather than each class's change since yesterday (that would need one more state file for little gain), and docker runs on each host from that host's own machine file instead of one central host list (each host's cleaner has its own in-use checks).

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions