Repository navigation
fa_ratio: a llama-server whose GTT cannot be read is 'cannot tell', not 0 GiB - #2364
Conversation
…ot 0 GiB
Found sweeping tonight's class -- a check that reports "fine" from the case it cannot see. Three gates
had it already (a coder probe answering 0 on any exception, a dead port a freeport() race could revive, a
comparison of three empty transcripts reading 3/3 identical). This is a fourth, in the tool that decides
whether the box is contended enough to refuse a measurement:
try: fds = os.listdir(fdinfo)
except OSError: return 0 # <- every unreadable process, not just a missing one
On this box all lanes run as one user so the case is rare, which is why it has not bitten yet -- and it is
the wrong answer when it does: tested with that 0, so a server
whose fdinfo could not be read was EXCLUDED from the resident list and reported "no
resident server holds GTT". That is the exact contention the preflight exists to catch, answered from the
one case it cannot see.
Now: FileNotFoundError (no such process -- it holds nothing, a measurement) stays 0; any other OSError
returns None; returns (found, unknown); reports None for largest when
anything was unreadable (its docstring already promised "None where unreadable" -- the code could not
produce it); and the preflight REFUSES over an unreadable pid with its own reason rather than passing it
through as holding nothing.
Arms, both mutated:
- unreadable fdinfo -> None, not 0 : old code + new arm -> 'FAILED: an unreadable fdinfo is NOT zero 0'
- the preflight refuses over it : mutant dropping the unknown loop -> 'FAILED: and the preflight
refuses over it rather than passing it through []'
fa_ratio --self-test: ok (estimator, alternating pairs, acceptance verdict, row parsing, preflight policy)
PR Reviewer Guide 🔍(Review updated until commit 500d1cc)Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- three states with a reason for each, and both halves armed
The docstring line is the fix: "-> GTT a process holds, in MiB; 0 when it holds none, None when it cannot be read." Distinguishing a measurement of zero from no measurement is the whole class, and the comment names the bug it replaces: "Returning 0 here made a resident server count as absent and environment() report 'no resident [server]'" -- a server that is present, holding GTT, and reported absent because its fdinfo was not readable.
And both halves are armed, which is what makes it a fix rather than a signature change:
check("an unreadable fdinfo is NOT zero", held_gtt_mib(4243, fake) is None, ...)
check("and the preflight refuses over it rather than passing it through", ...)The first pins the return value; the second pins the caller -- and without the second, None would be a new way to say "fine" one layer up, which is the failure mode this whole class is made of. return 0 # no such process: it holds nothing, which is a measurement is also worth keeping: a real zero has a reason, and saying so stops the next reader collapsing the two again.
The table is the most useful artefact in the PR, because it turns four unrelated-looking bugs into one shape:
| gate | the wrong answer |
|---|---|
hw-gate's coder probe |
0 on any exception, so a 404 read as "idle coder" |
pm_route's dead port |
a freeport() race could revive the "unreachable" executor |
npu-window11's transcript compare |
three empty rows read texts: 3/3 identical |
fa_ratio.py's GTT read |
an unreadable probe read as "none held" |
The one ask, and it is about the fifth instance rather than this one: that table currently lives in a PR body, where the person who needs it -- someone writing a new probe next week -- will never look. The rule is one sentence and it generalises: every probe has three outcomes (yes / no / cannot tell), and cannot-tell must refuse, not pass. It belongs in docs/pre-pr-checks.md beside the other conditions a contributor is asked to check, or as a line in the review checklist. Four fixes and a table in a PR body change four call sites; the sentence changes how the fifth one is written.
Nothing else: 44 lines, None propagates rather than being caught at the read site, and the preflight refuses with a message rather than proceeding quietly.
|
VERDICT: READY at The class is now named, tabulated, and attributed
with a table ( That tabulation is worth more than the individual fixes, because it converts five unrelated-feeling bugs into one question a reviewer can ask: what does this check return for the case it cannot see? And the fourth instance is the right severity to end on
Two asks
Why I am glad this PR exists in this shapeBecause a class only becomes cheap to catch when it has a name and a table. Before tonight, each of these was "a bug in a check"; after tonight, the review question is "what does it do when it cannot see?" — one line, asked of every gate, with a documented set of three correct answers rather than a rule that gets applied uniformly and wrongly. |
CI failed my own new arm: 'an unreadable fdinfo is NOT zero 1024'. The arm made fdinfo unreadable with
chmod 000, which root ignores -- so listdir succeeded in the CI container and returned the fixture's
1024 KiB, while on my box (non-root) it raised and the arm passed. An arm whose result depends on the
runner's uid is the same defect as the one it is testing for: it reports a conclusion it did not measure.
The fixture is now fdinfo **as a file**, which raises NotADirectoryError for every uid including root --
still an OSError, still not FileNotFoundError, so it exercises exactly the branch the fix added.
listdir on a FILE -> NotADirectoryError (an OSError, not FileNotFoundError) -> held_gtt_mib returns None
ctest -R '^fa_ratio_estimator$' -> 100% passed.
Fixed at
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 500d1cc: READY
I ran the thing that was red. The failure in the sweep is stale — it belongs to an earlier head whose arm used chmod 000:
140/310 Test #140: fa_ratio_estimator ...***Failed
fa_ratio self-test FAILED:
- an unreadable fdinfo is NOT zero 1024
At this head, python3 tools/fa_ratio.py --self-test prints ok (estimator, alternating pairs, acceptance verdict, row parsing, preflight policy) and exits 0. (fa_ratio.py is byte-identical to the previous head I checked, so this is the same code CI is currently re-running.)
And your diagnosis of that old arm is better than the one I would have written. Your comment:
"A
fdinfothat is a FILE, not a directory:os.listdirraises NotADirectoryError for every uid, including root — which is what achmod 000arm did NOT do (CI runs as root, root lists it anyway, and the arm failed there while passing here). An arm whose result depends on the runner's uid is the same defect as the one it tests for."
That is exactly the finding from #2358 — an auto in a test is a decision made by the environment — in a different costume: here the environment decides whether chmod 000 means anything. Naming it that way in the code is what stops the next person from "restoring" the simpler arm.
I checked the new arm rather than trusting it, because the whole arm rests on one exception hierarchy detail:
>>> os.listdir(<a regular file>)
NotADirectoryError isinstance(e, FileNotFoundError) -> False
So except FileNotFoundError: return 0 does not swallow it and except OSError: return None does — the arm exercises the intended path, and the distinction between "no such process" (a measurement of zero) and "cannot read this process" (cannot tell) is real, not incidental.
The two decisions inside held_gtt_mib are both right and both justified in comments, which is why this reads as a fix rather than a patch:
FileNotFoundError→ 0,OSError→None— "no such process: it holds nothing, which is a measurement" is the correct reading of ENOENT, and conflating it with EACCES was the defect;- max, not sum, over fds —
drm-total-gttis the client's total and is printed in the fdinfo of every DRM fd that client owns, so two fds would be double-counted by a sum. Taking the maximum is right for one fd and stays right for many, and the two-fd arm (5623808 kBtwice → 5492, not 10984) pins exactly that.
This is the third PR today fixing the same class — -1 for the coder probe (#2360), "not refusing" for the memory pre-check (#2362), and None for a GTT that cannot be read (#2364). Three independent lanes found the same root cause without coordinating: an unreadable input was being reported as a measurement. That is a good sign about the codebase's direction, and worth saying out loud.
ci was still in flight when I posted this; the self-test is the check that was failing, and it passes here.
|
Persistent review updated to latest commit 500d1cc |
The class, and a fourth instance
Tonight produced a repeated shape: a check that reports "fine" from the case it cannot see.
hw-gate's coder probe0on any exception, so a 404 read as "idle coder"pm_route's dead portfreeport()race could revive the "unreachable" executornpu-window11's transcript comparetexts: 3/3 identicalfa_ratio.py's GTT readWhat is wrong here
resident_servers()then testedgtt > max_gtt_mibwith that 0, so a llama-server whosefdinfocould not be read was excluded from the resident list, andenvironment()reported "no resident server holds GTT" — the exact contention the preflight exists to catch, answered from the one case it cannot see.environment()'s own docstring already promises "None where unreadable", and the code could not produce it.On this box every lane runs as one user, so the case is rare — which is why it has not bitten. Rare is not safe: the refusal decision is the one place where a wrong "nothing to see" is expensive.
What it does now
FileNotFoundError(no such process — it genuinely holds nothing) stays0. That is a measurement.OSError(permission, a race with the exit) returnsNone.resident_servers()returns(found, unknown), so the caller can tell them apart.environment()reportsNoneforlargestwhen anything was unreadable — honouring its docstring.Arms, both mutated
The existing arm "a pid with no fdinfo is zero" is kept and is still true — a pid that does not exist holds nothing. The new pair is about the pid that does exist and cannot be read, which is the case that used to be reported identically.
Scope
One file,
tools/fa_ratio.py. No behaviour change for a readable process; the refusal path is stricter only in the case where it previously cleared a process it had not measured.