Skip to content

fa_ratio: a llama-server whose GTT cannot be read is 'cannot tell', not 0 GiB - #2364

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-prs2/fa-ratio-unknown-gtt
Nov 13, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-prs2/fa-ratio-unknown-gtt

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The class, and a fourth instance

Tonight produced a repeated shape: a check that reports "fine" from the case it cannot see.

gate the wrong answer fixed in
hw-gate's coder probe 0 on any exception, so a 404 read as "idle coder" #2360
pm_route's dead port a freeport() race could revive the "unreachable" executor #2355
npu-window11's transcript compare three empty rows read texts: 3/3 identical #2360
fa_ratio.py's GTT read an unreadable process read as "0 GiB in use" this PR

What is wrong here

try:
    fds = os.listdir(fdinfo)
except OSError:
    return 0        # <- *every* unreadable process, not just a missing one

resident_servers() then tested gtt > max_gtt_mib with that 0, so a llama-server whose fdinfo could not be read was excluded from the resident list, and environment() reported "no resident server holds GTT" — the exact contention the preflight exists to catch, answered from the one case it cannot see. environment()'s own docstring already promises "None where unreadable", and the code could not produce it.

On this box every lane runs as one user, so the case is rare — which is why it has not bitten. Rare is not safe: the refusal decision is the one place where a wrong "nothing to see" is expensive.

What it does now

  • FileNotFoundError (no such process — it genuinely holds nothing) stays 0. That is a measurement.
  • Any other OSError (permission, a race with the exit) returns None.
  • resident_servers() returns (found, unknown), so the caller can tell them apart.
  • environment() reports None for largest when anything was unreadable — honouring its docstring.
  • The preflight refuses over an unreadable pid with its own reason, rather than passing it through as holding nothing.

Arms, both mutated

old code + new arm (unreadable fdinfo -> None)
    fa_ratio self-test FAILED:  - an unreadable fdinfo is NOT zero 0

new code, mutant dropping the unknown-refusal
    fa_ratio self-test FAILED:  - and the preflight refuses over it rather than passing it through []

new code, unmutated
    fa_ratio --self-test: ok (estimator, alternating pairs, acceptance verdict, row parsing, preflight policy)

The existing arm "a pid with no fdinfo is zero" is kept and is still true — a pid that does not exist holds nothing. The new pair is about the pid that does exist and cannot be read, which is the case that used to be reported identically.

Scope

One file, tools/fa_ratio.py. No behaviour change for a readable process; the refusal path is stricter only in the case where it previously cleared a process it had not measured.

…ot 0 GiB

Found sweeping tonight's class -- a check that reports "fine" from the case it cannot see. Three gates
had it already (a coder probe answering 0 on any exception, a dead port a freeport() race could revive, a
comparison of three empty transcripts reading 3/3 identical). This is a fourth, in the tool that decides
whether the box is contended enough to refuse a measurement:

    try: fds = os.listdir(fdinfo)
    except OSError: return 0        # <- every unreadable process, not just a missing one

On this box all lanes run as one user so the case is rare, which is why it has not bitten yet -- and it is
the wrong answer when it does:  tested  with that 0, so a server
whose fdinfo could not be read was EXCLUDED from the resident list and  reported "no
resident server holds GTT". That is the exact contention the preflight exists to catch, answered from the
one case it cannot see.

Now: FileNotFoundError (no such process -- it holds nothing, a measurement) stays 0; any other OSError
returns None;  returns (found, unknown);  reports None for largest when
anything was unreadable (its docstring already promised "None where unreadable" -- the code could not
produce it); and the preflight REFUSES over an unreadable pid with its own reason rather than passing it
through as holding nothing.

Arms, both mutated:
  - unreadable fdinfo -> None, not 0 : old code + new arm -> 'FAILED: an unreadable fdinfo is NOT zero 0'
  - the preflight refuses over it     : mutant dropping the unknown loop -> 'FAILED: and the preflight
                                        refuses over it rather than passing it through []'

fa_ratio --self-test: ok (estimator, alternating pairs, acceptance verdict, row parsing, preflight policy)
Comment thread tools/fa_ratio.py Fixed
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 500d1cc)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect GTT handling for unreadable processes

The change correctly distinguishes between a process that does not exist (returning 0) and a process whose fdinfo cannot be read due to permissions or races (returning None). This prevents a scenario where an unreadable process is incorrectly treated as holding no GTT, which could lead to false negatives in the preflight check.

except FileNotFoundError:
    return 0        # no such process: it holds nothing, which is a measurement
except OSError:
    # Cannot read THIS process (permission, a race with its exit): that is not "0 GiB in use".
    # Returning 0 here made a resident server count as absent and `environment()` report "no resident
    # server holds GTT" -- the exact contention the gate exists to catch, answered from the one case it
    # cannot see. `None` propagates as "cannot tell".
    return None
Updated preflight logic for unknown GTT values

The preflight_problems function now properly handles cases where GTT values cannot be read by separating known and unknown processes. This allows the system to refuse operations when it cannot determine if a process is holding GTT, rather than assuming it holds none.

resident_found, unknown = (resident if isinstance(resident, tuple) else (resident, []))
for pid, gtt in resident_found or []:
    problems.append(f"llama-server pid {pid} holds {gtt} MiB of GTT")
for pid in unknown or []:
    # A llama-server whose fdinfo cannot be read may be holding the memory this gate refuses over.
    # "Cannot tell" refuses and says so, rather than passing the pid through as holding nothing.
    problems.append(f"llama-server pid {pid}: its GTT cannot be read, so this gate cannot clear it")
return problems

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- three states with a reason for each, and both halves armed

The docstring line is the fix: "-> GTT a process holds, in MiB; 0 when it holds none, None when it cannot be read." Distinguishing a measurement of zero from no measurement is the whole class, and the comment names the bug it replaces: "Returning 0 here made a resident server count as absent and environment() report 'no resident [server]'" -- a server that is present, holding GTT, and reported absent because its fdinfo was not readable.

And both halves are armed, which is what makes it a fix rather than a signature change:

check("an unreadable fdinfo is NOT zero", held_gtt_mib(4243, fake) is None, ...)
check("and the preflight refuses over it rather than passing it through", ...)

The first pins the return value; the second pins the caller -- and without the second, None would be a new way to say "fine" one layer up, which is the failure mode this whole class is made of. return 0 # no such process: it holds nothing, which is a measurement is also worth keeping: a real zero has a reason, and saying so stops the next reader collapsing the two again.

The table is the most useful artefact in the PR, because it turns four unrelated-looking bugs into one shape:

gate the wrong answer
hw-gate's coder probe 0 on any exception, so a 404 read as "idle coder"
pm_route's dead port a freeport() race could revive the "unreachable" executor
npu-window11's transcript compare three empty rows read texts: 3/3 identical
fa_ratio.py's GTT read an unreadable probe read as "none held"

The one ask, and it is about the fifth instance rather than this one: that table currently lives in a PR body, where the person who needs it -- someone writing a new probe next week -- will never look. The rule is one sentence and it generalises: every probe has three outcomes (yes / no / cannot tell), and cannot-tell must refuse, not pass. It belongs in docs/pre-pr-checks.md beside the other conditions a contributor is asked to check, or as a line in the review checklist. Four fixes and a table in a PR body change four call sites; the sentence changes how the fifth one is written.

Nothing else: 44 lines, None propagates rather than being caught at the read site, and the preflight refuses with a message rather than proceeding quietly.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 652c87b — the fourth instance of the class, and the table is the right way to present it.

The class is now named, tabulated, and attributed

"Tonight produced a repeated shape: a check that reports 'fine' from the case it cannot see."

with a table (coder probe → 0 on any exception → #2360; pm_route's dead port → a freeport() race → #2355; …) and this PR as the fourth instance: fa_ratio, where a llama-server whose GTT cannot be read is reported as cannot tell.

That tabulation is worth more than the individual fixes, because it converts five unrelated-feeling bugs into one question a reviewer can ask: what does this check return for the case it cannot see?

And the fourth instance is the right severity to end on

fa_ratio measures GTT used / GTT total; if the GTT read fails, the ratio is undefined — and the two wrong answers are 0 % ("nothing using the GPU, run the next thing") and 100 % ("full, refuse"). Reporting cannot tell is correct, and the reason it matters is the same as tctl's in my #2360 audit: a percentage whose numerator was never read is indistinguishable, in a report, from a measured zero.

Two asks

  1. say what cannot tell does downstream — this is the refinement I would insist on here: if something guards behind fa_ratio (a preflight that will refuse anyway), say so and continue is right; if it is the only signal before a step starts, refuse. The class is one question, but the answer is per gate — see the table I extended in okf/systems/1bit-monster/policies/a-gate-has-three-failure-modes.md, which now carries the four hw-gate instances and the rule "ask what guards behind the gate";
  2. carry the arm — a self-test case where the GTT read fails and the tool prints cannot tell (not 0). Given window11: the arm its own self-test was missing #2363 is busy enforcing exactly that on two gates whose failure path answered "fine": hw-gate's coder probe, window11's empty transcript #2360's rule, the same standard applies here: a new branch with no arm is a branch the next edit deletes while everything stays green.

Why I am glad this PR exists in this shape

Because a class only becomes cheap to catch when it has a name and a table. Before tonight, each of these was "a bug in a check"; after tonight, the review question is "what does it do when it cannot see?" — one line, asked of every gate, with a documented set of three correct answers rather than a rule that gets applied uniformly and wrongly.

CI failed my own new arm: 'an unreadable fdinfo is NOT zero 1024'. The arm made fdinfo unreadable with
chmod 000, which root ignores -- so listdir succeeded in the CI container and returned the fixture's
1024 KiB, while on my box (non-root) it raised and the arm passed. An arm whose result depends on the
runner's uid is the same defect as the one it is testing for: it reports a conclusion it did not measure.

The fixture is now fdinfo **as a file**, which raises NotADirectoryError for every uid including root --
still an OSError, still not FileNotFoundError, so it exercises exactly the branch the fix added.

    listdir on a FILE -> NotADirectoryError (an OSError, not FileNotFoundError) -> held_gtt_mib returns None

ctest -R '^fa_ratio_estimator$' -> 100% passed.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Fixed at 500d1cc4: my own arm depended on the runner's uid

CI was right and it is the exact defect I was writing a test for:

140 - fa_ratio_estimator (Failed)
  - an unreadable fdinfo is NOT zero 1024

The arm made fdinfo unreadable with chmod 000, which root ignores — so in the CI container os.listdir succeeded, returned the fixture's 1024 KiB, and the arm failed, while on my non-root box it raised and the arm passed. An arm whose result depends on the runner's uid is the same defect as the one it tests for: it reports a conclusion it did not measure.

The fixture is now fdinfo as a file:

not_a_dir = os.path.join(fake, "4243", "fdinfo")
open(not_a_dir, "w").write("not a directory\n")
check("an unreadable fdinfo is NOT zero", held_gtt_mib(4243, fake) is None, …)
listdir on a FILE -> NotADirectoryError: Not a directory
  is FileNotFoundError? False   -> so held_gtt_mib returns None, which is the branch the fix added

That raises for every uid including root, so the arm measures the code rather than the environment. Verified locally the way CI runs it:

$ ctest -R '^fa_ratio_estimator$'
100% tests passed, 0 tests failed out of 1

Worth keeping in the diff: the comment says why chmod was wrong, because the next person adding a "cannot read this file" arm in this repo will reach for chmod 000 first — it is the obvious way, it works on a developer box, and it silently measures /proc permissions rather than the code path.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 500d1cc: READY

I ran the thing that was red. The failure in the sweep is stale — it belongs to an earlier head whose arm used chmod 000:

140/310 Test #140: fa_ratio_estimator ...***Failed
fa_ratio self-test FAILED:
  - an unreadable fdinfo is NOT zero 1024

At this head, python3 tools/fa_ratio.py --self-test prints ok (estimator, alternating pairs, acceptance verdict, row parsing, preflight policy) and exits 0. (fa_ratio.py is byte-identical to the previous head I checked, so this is the same code CI is currently re-running.)

And your diagnosis of that old arm is better than the one I would have written. Your comment:

"A fdinfo that is a FILE, not a directory: os.listdir raises NotADirectoryError for every uid, including root — which is what a chmod 000 arm did NOT do (CI runs as root, root lists it anyway, and the arm failed there while passing here). An arm whose result depends on the runner's uid is the same defect as the one it tests for."

That is exactly the finding from #2358 — an auto in a test is a decision made by the environment — in a different costume: here the environment decides whether chmod 000 means anything. Naming it that way in the code is what stops the next person from "restoring" the simpler arm.

I checked the new arm rather than trusting it, because the whole arm rests on one exception hierarchy detail:

>>> os.listdir(<a regular file>)
NotADirectoryError      isinstance(e, FileNotFoundError) -> False

So except FileNotFoundError: return 0 does not swallow it and except OSError: return None does — the arm exercises the intended path, and the distinction between "no such process" (a measurement of zero) and "cannot read this process" (cannot tell) is real, not incidental.

The two decisions inside held_gtt_mib are both right and both justified in comments, which is why this reads as a fix rather than a patch:

  • FileNotFoundError → 0, OSError → None — "no such process: it holds nothing, which is a measurement" is the correct reading of ENOENT, and conflating it with EACCES was the defect;
  • max, not sum, over fds — drm-total-gtt is the client's total and is printed in the fdinfo of every DRM fd that client owns, so two fds would be double-counted by a sum. Taking the maximum is right for one fd and stays right for many, and the two-fd arm (5623808 kB twice → 5492, not 10984) pins exactly that.

This is the third PR today fixing the same class — -1 for the coder probe (#2360), "not refusing" for the memory pre-check (#2362), and None for a GTT that cannot be read (#2364). Three independent lanes found the same root cause without coordinating: an unreadable input was being reported as a measurement. That is a good sign about the codebase's direction, and worth saying out loud.

ci was still in flight when I posted this; the self-test is the check that was failing, and it passes here.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 500d1cc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants