desktop: Training state is memory-only: a restart loses the live run, and "one GPU runs one job" stops refusing

[MervinPraison]

Found by a six-lens audit of the desktop app on 27 Aug 2026. Every finding was reproduced against the code, not inferred.
Severity: high · audit rank 3 of 16.

Breaks: after any engine restart (crash, relaunch, the kill in #12), /train/runs returns [] while the run directory sits on disk, /train/stop answers nothing was running, and starting a second fine-tune succeeds — two trainers on one GPU, which Trainer.start's own docstring (engine/training.py:230) says is an OOM that must be refused. ui/index.html:1357 claims the opposite: "a fine-tune started in an earlier session reappears rather than looking like it never happened."
Who / likelihood: anyone who restarts the app during a run. Reproduced: two live trainers, one of them an orphan with ppid 1.
Where: engine/training.py:221,224 — self.current/self.history are built empty and never reconstructed from self.dir. Only config.yaml and train.log are on disk; there is no state record at all.

This is the largest fix on the list, and nothing smaller produces the effect — the information simply is not persisted:

--- a/engine/training.py
+++ b/engine/training.py
@@ -222,6 +222,52 @@ class Trainer:
         self.history = collections.deque(maxlen=MAX_HISTORY)
         self._lock = threading.Lock()
+        self._reload()
+
+    # -- surviving a restart --------------------------------------------
+    def _state_path(self, run_id):
+        return os.path.join(self.dir, run_id, "run.json")
+
+    def _persist(self, run):
+        """Enough on disk to answer 'is a job on the GPU right now'."""
+        try:
+            with open(self._state_path(run.id), "w", encoding="utf-8") as fh:
+                json.dump({**run.summary(), "pid": run.pid}, fh)
+        except OSError:
+            pass
+
+    def _reload(self):
+        """Rebuild history, adopting a run whose process is still alive.
+
+        A run the new process cannot see is not 'missing' -- it is either
+        still on the GPU (adopt it, so stop() can reach it) or it was
+        interrupted (say so). Reporting neither is how a second fine-tune
+        got started beside a live one.
+        """
+        for name in sorted(self._existing_ids()):
+            try:
+                with open(self._state_path(name), encoding="utf-8") as fh:
+                    saved = json.load(fh)
+            except (OSError, ValueError):
+                continue
+            run = Run(name, os.path.join(self.dir, name, "config.yaml"),
+                      os.path.join(self.dir, name, "train.log"))
+            run.state = saved.get("state", FAILED)
+            run.started, run.ended = saved.get("started", 0), saved.get("ended")
+            run.error, run.pid = saved.get("error"), saved.get("pid")
+            if run.state not in TERMINAL:
+                if run.pid and _pid_alive(run.pid):
+                    run.state = RUNNING
+                    self.current = run          # stop() signals run.pid's group
+                else:
+                    run.finish(FAILED,
+                               "the engine restarted while this run was live")
+            self.history.appendleft(run)

with (a) Run.__init__ gaining self.pid = None, (b) _persist(run) called after run._proc = proc in _supervise, after the RUNNING transition, and inside finish, (c) stop() falling back from run._proc to run.pid via a _terminate_pid_group(pid) sibling of the existing _terminate_group(proc), and (d) a three-line _pid_alive (os.kill(pid, 0) on POSIX, OpenProcess/tasklist on Windows — or reuse the shell's observe-style start-time check if you want PID-reuse safety, which this minimal version does not have).

Fixing #2 shrinks the window; it does not close it, because a crash or a SIGKILL leaves the same state.

Test — engine/test_training.py (already runs real subprocesses and real SIGTERM): start a run on a stub trainer that sleeps; construct a second Trainer over the same home; assert (a) history contains the run, (b) start() on the second trainer raises RuntimeError, (c) stop() on the second trainer actually kills the first trainer's pid. All three fail today.

---

---

Not yet fixed. Filed so it is not lost with the session that found it. The fix and the test above are proposals from the audit — worth re-checking against current main before implementing, since the file has moved since.

[praisonai-triage-agent]

Claude finished @MervinPraison's task —— View job • claude/issue-4492-20260827-1338 • [Create PR ➔](https://github.com/MervinPraison/PraisonAI/compare/main...claude/issue-4492-20260827-1338?quick_pull=1&title=Issue%20%234492%3A%20Changes%20from%20Claude&body=This%20PR%20addresses%20issue%20%234492%0A%0AGenerated%20with%20%5BClaude%20Code%5D(https%3A%2F%2Fclaude.ai%2Fcode)

---

Training state now survives a restart — done

PR: #4510

• Read AGENTS.md guidelines
• Locate engine/training.py and related files
• Understand current Trainer/Run + server.py usage
• Validate architecture routing (desktop engine, stdlib-only — in scope, no new deps)
• Implement fix (persist run state, reload on restart, adopt live PID)
• Update tests
• Run tests — 60 passed
• Commit, push, create PR

What changed (src/praisonai-desktop/engine/training.py)

The subsystem kept the live run and history in memory only, so any engine restart lost the run: /train/runs returned [], /train/stop had nothing to stop, and start() launched a second trainer beside a live one — the OOM that "one GPU runs one job" exists to refuse.

• Run.pid records the child's pid (outlives the restart; _proc does not).
• Trainer._persist(run) writes a per-run run.json (summary() + pid) at spawn, on the RUNNING/STOPPING transition, and on finish.
• Trainer._reload() (in __init__) rebuilds history: a run whose pid is alive is adopted (current set, so start refuses and stop reaches it); an interrupted run whose pid is gone is marked failed.
• stop() falls back from run._proc to _terminate_pid_group(run.pid) for an adopted run and records the ending itself.
• _pid_alive / _terminate_pid_group: POSIX os.kill(pid,0) / killpg, Windows OpenProcess / taskkill /T, with the same "never signal our own group" guard.

Tests (test_training.py — new SurvivingARestart)

A second Trainer over the same home stands in for the process after a restart: a live run reappears as running; a second fine-tune is refused; stop() kills the adopted pid → cancelled; an interrupted run reads failed (not missing) and isn't adopted; a finished run isn't re-adopted.

Kept lightweight and stdlib-only — no new params/deps. The pre-existing ResourceWarnings in the test output come from unclosed open() calls already in the file, unrelated to this change.

Job run · Branch