Found by a six-lens audit of the desktop app on 27 Aug 2026. Every finding was reproduced against the code, not inferred.
Severity: high · audit rank 4 of 16.
Breaks: one OSError on the log write abandons the reader while the child is still writing to a PIPE nobody drains. The child blocks on the full pipe, proc.wait() blocks on the child, run.finish() never runs — the run reads running permanently and Trainer.start refuses every subsequent run for the life of the engine. One [reader stopped: …] line scrolls past and nothing else is said.
Who / likelihood: a full disk, a revoked directory, a network home — uncommon, but the failure is permanent and needs a restart the user has no reason to connect to it. Reproduced with r5.py (child would print ~1 MB and exit in under a second; first log write ENOSPC): +6s state=running child alive=True events=3.
Where: engine/training.py:355-361.
--- a/engine/training.py
+++ b/engine/training.py
@@ -353,11 +353,29 @@ class Trainer:
- try:
- with open(run.log_path, "a", encoding="utf-8") as log:
- for line in proc.stdout:
- log.write(line)
- self._consume(run, line.rstrip("\n"))
- except Exception as exc: # noqa: BLE001
- run.emit("log", {"line": f"[reader stopped: {exc}]"})
+ try:
+ log = open(run.log_path, "a", encoding="utf-8")
+ except OSError as exc:
+ log = None
+ run.emit("log", {"line": f"[log unavailable: {exc}]"})
+ try:
+ for line in proc.stdout:
+ # The pipe is drained whatever happens to the log. Stopping
+ # the loop leaves the child blocked on a full pipe and
+ # proc.wait() blocked on the child, so the run never ends
+ # and every later run is refused.
+ if log is not None:
+ try:
+ log.write(line)
+ log.flush() # the log is the full record only if it exists
+ except OSError as exc:
+ run.emit("log", {"line": f"[log stopped: {exc}]"})
+ log.close()
+ log = None
+ try:
+ self._consume(run, line.rstrip("\n"))
+ except Exception as exc: # noqa: BLE001
+ run.emit("log", {"line": f"[reader error: {exc}]"})
+ finally:
+ if log is not None:
+ log.close()
code = proc.wait()The flush() also fixes the separate observation that train.log is block-buffered and never flushed — 0 bytes on disk throughout a live run, and 0 bytes after a kill, while training.py:37 and :156 cite "the log file is the full record" as the reason for capping in-memory events at 4000.
Test — engine/test_training.py: stub trainer prints ~200 KB then exits 0; patch the log open (or point log_path at a directory) so writing fails; assert within a few seconds that run.state == DONE and train.log's absence did not stop metrics/events from growing past the pre-failure count. Today the state stays running indefinitely.
Not yet fixed. Filed so it is not lost with the session that found it. The fix and the test above are proposals from the audit — worth re-checking against current main before implementing, since the file has moved since.
Breaks: one
OSErroron the log write abandons the reader while the child is still writing to aPIPEnobody drains. The child blocks on the full pipe,proc.wait()blocks on the child,run.finish()never runs — the run readsrunningpermanently andTrainer.startrefuses every subsequent run for the life of the engine. One[reader stopped: …]line scrolls past and nothing else is said.Who / likelihood: a full disk, a revoked directory, a network home — uncommon, but the failure is permanent and needs a restart the user has no reason to connect to it. Reproduced with
r5.py(child would print ~1 MB and exit in under a second; first log write ENOSPC):+6s state=running child alive=True events=3.Where:
engine/training.py:355-361.The
flush()also fixes the separate observation thattrain.logis block-buffered and never flushed — 0 bytes on disk throughout a live run, and 0 bytes after a kill, whiletraining.py:37and:156cite "the log file is the full record" as the reason for capping in-memory events at 4000.Test —
engine/test_training.py: stub trainer prints ~200 KB then exits 0; patch the log open (or pointlog_pathat a directory) so writing fails; assert within a few seconds thatrun.state == DONEandtrain.log's absence did not stopmetrics/eventsfrom growing past the pre-failure count. Today the state staysrunningindefinitely.Not yet fixed. Filed so it is not lost with the session that found it. The fix and the test above are proposals from the audit — worth re-checking against current
mainbefore implementing, since the file has moved since.