Skip to content

fix: the audit follow-ups, and the three limits that remain - #23

Merged
Odiethebest merged 5 commits into
mainfrom
fix/audit-followups
Aug 3, 2026
Merged

Odiethebest merged 5 commits into
mainfrom
fix/audit-followups

Conversation

@Odiethebest

Copy link
Copy Markdown
Owner

Stacked on #18–#22. The last PR in the deployment series. Eight findings, 40 new tests, three commits grouped by what they are about rather than by finding number.

Finding What changed
F-03 Request bodies had no size limit anywhere. query had a min_length and no max; history was trimmed only after parsing
F-04 API and agent both refuse to start on the published placeholder key
F-06 Resolution checked, credentials refused, optional host allowlist
F-07 Tool cache keyed by index_revision
F-08 Optional oldest-first workdir cap, off by default
F-10 /readyz, including the embedding-dimension check
F-14 -- before the clone URL, GIT_TERMINAL_PROMPT=0
F-15 grep timeout, match caps, file-size cap

The ones worth reading

F-07 was invisible. The tool cache key was (tool, repo_id, args) and carried nothing about which generation of the corpus produced the answer, so re-indexing left every cached result addressable for 24 hours and the agent answered from the previous corpus. Nothing failed, nothing logged. index_revision is now keyword-only with no default — a default is what would let a new call site quietly reintroduce it. Read once per query so one walk sees one corpus; when it cannot be read the request bypasses the cache rather than falling back to a revision-less key, because that fallback recreates the collision in exactly the case nobody exercises.

F-06's real gap was names. The old check only ever looked at literal IPs, so evil.example.com pointing at 169.254.169.254 passed. Credentials in the URL are refused rather than stripped — a submitted URL is persisted and echoed back by the status route, so a token handed to this endpoint became readable by anyone who could read that repository's status. Quietly rewriting what someone submitted is the worse behaviour.

Resolution goes through a module seam so the suite performs no real DNS lookups; without it every test that submits a URL would need the network.

F-10 also catches the one silent misconfiguration. /readyz compares the configured EMBEDDING_DIM against the live column — the failure §5.4 describes, where the worker's inserts fail or dense search runs a stub all-zero query vector against real vectors and returns noise, with no error either way. A missing chunks table is skipped rather than failed, because the Railway runbook starts the API before the migration.

/healthz is left shallow on purpose: a liveness probe that fails on a dependency gets the process restarted, which fixes nothing and loses the logs.

Three limits this PR does not remove

Stated because a closed finding that quietly leaves the hole open is worse than an open one. All three stay visible in the register rather than struck through.

  1. DNS rebinding is not closed. Every resolved address is checked, but git resolves the name again when it clones. The fix is egress filtering at the network — on Railway a platform capability this project does not configure. REPO_URL_ALLOWED_HOSTS is the available mitigation.
  2. The workdir cap is a trade, not a cleanup. The agent reads those trees at query time, so evicting a ready repository's checkout costs it read_file / grep / list_directory until re-index, while search and the call graph carry on from Postgres. Off by default for that reason, and the tool's error now says which half stopped working instead of raising a bare FileNotFoundError. The durable answer is Deploy.md §11 item 2.
  3. The grep Python fallback can still block. Python cannot interrupt a single re call, so catastrophic backtracking against one long line is unbounded. Everything countable is capped, and the agent image installs ripgrep so the path runs only where it is absent.

Unchanged on purpose

Docker health checks still probe /healthz. /readyz is for a load balancer; wiring it into depends_on: condition: service_healthy would let an unrelated dependency outage block local startup chains. The runbook says to point Railway's health check at /readyz.

make check green: 394 passed / 5 skipped, 86 frontend tests, mypy strict over 88 files.

… key

Four of the audit follow-ups, grouped because they are all "an input this
service trusted more than it should".

Request bodies had no size limits anywhere. QueryRequest.query had a
min_length and no max, so a caller could make the gateway parse an
arbitrarily large body into memory before anything rejected it; history was
trimmed only *after* parsing. Bounds are ceilings on abuse, not on use — no
Git URL approaches 2 KB and the gateway already trims turns to 4,000
characters.

git clone received the repository URL with no `--`, so a value beginning
with "-" is parsed as an option and `--upload-pack=<cmd>` is command
execution. The gateway validates the URL, but the worker consumes a queue
message and should not depend on the only check being upstream of it. Also
sets GIT_TERMINAL_PROMPT=0: git otherwise waits for a username on a private
or mistyped repository until the 180s clone timeout, when "authentication
required" after two seconds is the better answer.

INTERNAL_API_KEY had a working default published in this repository, and
nothing but compose's `${VAR:?}` stopped a service running on it — which
covers compose and not a container started straight from the Dockerfile,
i.e. what a platform deploy does. Both the API and the agent now refuse to
start on the placeholder. The additional length rule applies only when the
deployment authenticates its users, which is a proxy for production rather
than a fact about it; that is stated in the code rather than implied.

The SSRF check only ever looked at literal IPs, so a hostname resolving to
cloud metadata or an internal subnet passed. Every resolved address is now
checked, one private answer among several public ones is enough to refuse,
and a resolution failure does *not* reject — a DNS hiccup is not evidence
about the host, and reporting it as an invalid URL would send someone
hunting for a typo. Credentials in the URL are refused rather than
stripped: a submitted URL is persisted and echoed back by the status route,
so a token handed to this endpoint became readable by anyone who could read
that repository's status. And there is now an optional host allowlist,
empty by default, which is the only rule in the whole check that states
what is *allowed* rather than what is obviously wrong.

What resolution does not close is rebinding — git resolves the name again
when it clones. That needs egress filtering at the network, and it is
recorded as unclosed rather than implied to be handled.

Resolution goes through a module-level seam so the tests do not perform
real DNS lookups; an autouse fixture stubs it, and the tests that care
about resolution replace it. Without that the suite would need the network
and would occasionally fail for reasons unrelated to the code.
The tool cache key was (tool, repo_id, args) and carried nothing about
which generation of the corpus produced the answer. Re-indexing a
repository therefore left every cached result addressable by the same key
for the next 24 hours, and the agent answered from the previous corpus.
Nothing failed, nothing logged, and the only symptom was answers that
disagreed with the index.

The worker already increments repos.index_revision in the same transaction
that replaces the chunks, so folding it into the key makes a re-index a
cache miss by construction rather than by remembering to flush Redis —
which is currently a documented manual step before evaluation runs.

`index_revision` on tool_cache_key is keyword-only with no default. A
default is what would let a new call site quietly reintroduce this.

Read once per query and memoised on the state, not per tool call, so one
walk sees one corpus even if a re-index lands mid-answer. When it cannot be
read — no session, no such repository, a query error — the request bypasses
the cache entirely instead of falling back to a revision-less key, because
that fallback would recreate the collision in exactly the case nobody
exercises.

Three tests cover it: same generation hits, a bumped generation misses, and
an unreadable revision produces no cache entry at all. An existing caching
test needed a session stub, which is itself the change being made visible.
The remaining three audit follow-ups, each with a limit stated rather than
implied.

grep had no ceiling of any kind. Its argument is a pattern the planner
derives from a user question, and `.` over a large repository produced
hundreds of MB of JSON buffered whole in the agent process, then serialised
into Redis and the SSE state. There is now a timeout (the clone path had one
and this did not), per-file and total match caps, and a file-size cap —
bounded at ripgrep rather than after the fact, since reading everything and
discarding most of it is the behaviour being replaced.

The Python fallback is bounded on everything countable and is honest about
the rest: Python cannot time out a single `re` call, so a pattern with
catastrophic backtracking against one long line can still block the loop.
The agent image installs ripgrep, so that path runs only where it is
missing. The docstring says so instead of implying the fallback is safe.

/healthz answered `ok` with Postgres, Redis and RabbitMQ all down, and the
first user request discovered it. /readyz checks all three — all of them,
not short-circuited, because an operator wants to know whether one thing is
broken or everything is. It stays a 503 rather than a 200 with a body: a
load balancer reads the status line.

It also checks the embedding dimension against the live column, which is
the one misconfiguration in this stack that produces no error at all
(Deploy.md §5.4): the worker's inserts fail, or dense search runs a stub
all-zero query vector against real stored vectors and returns noise. A
missing table is skipped rather than failed, because before the first
migration there is nothing to disagree with and the Railway runbook starts
the API before running it. /healthz is left shallow on purpose — a liveness
probe that fails on a dependency gets the process restarted, which fixes
nothing and loses the logs.

Cloned workdirs were never removed, for any repository, ever, so disk grew
monotonically with the number of distinct repositories ever submitted.
Eviction is a trade, not a cleanup: the agent reads those trees at query
time, so pruning a `ready` repository's checkout costs it read_file, grep
and list_directory until re-index while search and the call graph carry on
from Postgres. So it is a count cap, off by default, evicting oldest-first
before a clone and never the repository being cloned. An undeletable
directory is logged and skipped rather than failing the index.

And because that failure is now reachable, the agent's missing-checkout
error says what happened: a bare FileNotFoundError sends the reader looking
for a missing file rather than a missing checkout.
Eight findings closed or narrowed, with the register updated from what was
observed. Three of them are marked "mostly" rather than fixed, and the
reasons are written out rather than left in a commit message:

DNS rebinding survives resolution checking, because git resolves the name
again when it clones. That needs egress filtering at the network, which on
Railway is a platform capability this project does not configure.

The workdir cap is a trade. The agent reads those trees at query time, so
evicting a ready repository's checkout costs it read_file, grep and
list_directory until re-index while search and the call graph carry on from
Postgres. Off by default for that reason.

The grep Python fallback can still block on one pathological re call,
because Python offers no way to interrupt one. Everything countable is
capped, and the agent image installs ripgrep so the path runs only where it
is absent.

A closed finding that quietly leaves the hole open is worse than an open
one, so all three stay visible in the register rather than being struck
through.

The runbook also now says to point Railway's health check at /readyz rather
than /healthz, and to decide WORKDIR_MAX_REPOS and REPO_URL_ALLOWED_HOSTS
deliberately instead of inheriting defaults chosen to preserve existing
behaviour.
Three config-as-code files and the variable sheet that cannot be one.

TOML rather than JSON because several of these settings need a reason
attached and JSON has nowhere to put one. numReplicas = 1 on agent-worker is
the clearest case: a second replica would not share the volume, so its agent
would read a different or empty checkout, and it would also be a second
RabbitMQ consumer competing for jobs in a pipeline that is not written to run
concurrently on one repository. A bare `1` invites someone to raise it.

The migration becomes a preDeployCommand on api rather than a manual step.
That is where it belongs, not where it is convenient: it reads EMBEDDING_DIM
to size the pgvector column and that variable lives on the api service, so
running it anywhere else is how the column ends up at a dimension nothing
agrees with. It also fixes the ordering the runbook previously had to
describe in prose — EMBEDDING_DIM must now simply be right before the first
deploy of api.

api's health check is /readyz, not /healthz. The shallow one answers ok with
every dependency down.

Also unifies the frontend build context to the repository root. It was the
only image built from a subdirectory, which is awkward on a platform whose
config-file path does not follow the service's Root Directory — the
alternative was one service with a Root Directory and a `..` in its
dockerfilePath. Removing the apps/frontend exclusion from .dockerignore
costs almost nothing: node_modules and dist stay excluded, which is where
the 227 MB was. Verified by building: 1.47 MB of context, and the image
behaves identically — gzip, cache headers, SPA fallback, and both
envsubst shapes with nginx's own variables intact.

Still nothing deployed. The files and the runbook are written from the code
and say so.
@Odiethebest
Odiethebest merged commit 80cd204 into main Aug 3, 2026
1 of 2 checks passed
Odiethebest added a commit that referenced this pull request Aug 3, 2026
CI on main has been red since #21 and I did not look. Every report of
"make check green" in this series was true and insufficient: it passes
locally because a developer's .env supplies a real INTERNAL_API_KEY and a
Postgres is running. CI has neither, and CI is the hermetic environment that
exists to catch exactly this.

Two failures, both mine.

#21 added a test that asserted a protected route stops returning 401 by
hitting /repos/{id}/status, which connects to Postgres. The claim being
tested is "the cookie got past require_session", which is about the
dependency and not about the database. It now stubs the session and asserts
404 for an unknown repo — the request reached the handler, which is the
actual proof.

#23 added a startup guard refusing the placeholder INTERNAL_API_KEY, so on a
machine with no .env every test that boots a lifespan died with "still the
placeholder". conftest.py now pins it alongside AUTH_ENABLED and
DOCS_ENABLED. That file was written for precisely this asymmetry, and pinning
two settings and not the third left the hole open — the guard just made it
fatal instead of silent.

Verified in both environments rather than one: 394 passed with .env present,
and 394 passed with .env moved aside and DATABASE_URL pointed at a dead
port, which is the closest local reproduction of CI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant