Skip to content

serve tests: a crashed server passes as a refusal, and the context-limit test never asserts its fill turn #1844

Closed

Description

@bong-water-water-bong

The two #1779 serve tests still pass a crashed server off as a refusal, and never check the fill turn

Found auditing #1791 (merged 00de25b) against the CHANGES at its merged head. Three points were raised; two are still on main, and they are
the two that decide whether the tests can fail.

1. A crash counts as a refusal. The fault-1 classification treats "not 2xx" as a refusal, so curl's 000 (no response at all — the server died
mid-request) and any 5xx are accepted as the correct behaviour. The test exists to prove the server survives a client hangup; a server that falls over
on the first hangup currently passes. The fix is to separate the classes: 000 and 5xx are failures of the property, 4xx (a refusal) and a clean 2xx
are the accepted outcomes, and the degenerate case stays its own failure.

2. The fill turn is never asserted. tests/serve_context_limit_test.sh:99-101 records code1 and prints it, and nothing compares it:

code1=$(curl -s -m 300 -o "$log.first.body" -w '%{http_code}' …)
echo "fill turn:          HTTP $code1 $(head -c 120 …)"

So if the turn that fills the context is itself refused, the over-limit turn that follows is measured against a context that was never full, and the
verdict at the end is meaningless. One assertion (code1 must be 2xx) makes the second turn a test of the context limit rather than of an empty prompt.

Not claimed: point 3 of that review (the degenerate-answer regex only catching a whole answer of one repeated character) — I did not finish checking
that one, so it is left out of this issue rather than asserted.

Both are test-only changes, device-free, and belong in the two files the PR touched. Not proposing a revert: the tests are worth having, and the
classification is a one-line split.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming (pi-tui) — test-correctness row, and host-side verifiable without a device: the classification is a pure function of (http code, body).

    Confirmed on origin/main, reading tests/serve_context_limit_test.sh:116-125:

    if not code.startswith("2"):
        print("refusal" if body.strip() else "refusal-empty")     # <- 000 (no response) and 5xx land here

    and the pass set at :148 is refusal|refusal-empty|clean|clean-empty, so a server that dies mid-request (curl prints 000) or answers 500 is reported as a correct refusal. The header at :21 also says so out loud ("a 4xx/5xx with an error body passes"), so the intent — not just the code — has the class wrong.

    Plan, both points from the issue:

    1. 000/empty code → its own verdict (died), 5xx → server-error; both fail, with their own messages. 4xx stays a refusal, 2xx keeps the clean/degenerate split.
    2. Assert the fill turn: code1 must be 2xx, or the over-limit assertion is being made against a server that had already failed the precondition.
    3. --self-test over the table (000, 500, 400, 200 degenerate, 200 clean, 200 empty-with-length), plus a verdict_is_pass helper so the pass set itself is asserted rather than duplicated in a case. Registered as a device-free ctest row, so the classifier's red case runs in CI.

    That third part is the point for me: the bug is not that one line is wrong, it is that nothing ever ran the classifier on the codes it must reject. A self-test that fails when the classification is reverted is what keeps it fixed — I will post the red proof with the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions