CI trial 3 (do not merge)

CHANGELOG.md

@@ -262,9 +262,27 @@ Overview, how to try it and the known limits: docs/autonomy.md.
   itself, at `agents/<id>/memory`, at a layer or `emotion/` directory, at `index.sqlite3`
   or at an entry file is still refused, also when swapped in later; the per-read check now
   starts at the root, which halves the `lstat` calls of a read (#296).
+- **Memory store (cost of a read)** — the memory root is resolved once when the store is
+  built and the entry enumeration no longer walks every entry's components again: a read
+  no longer calls `Path.resolve()` at all (the link walk from the resolved root already
+  proves the entry is inside it). Measured on the proximity-recall evaluation: 1,171,359
+  `lstat` and 117,124 `resolve` calls become 10,097 and 986 (the whole test 27.6 s → 24.7 s
+  on macOS, where both are cheap; Windows pays far more per call). The no-follow open, the
+  identity compare and the component walk before and after the open are unchanged, and a
+  link swapped in just before the open — on the entry or on a parent — is still refused
+  (two new tests) (#662).
+- **PDF extraction** — a memory limit of zero or less is no longer taken as an enforced
+  limit: Linux accepts a negative `RLIMIT_AS` as "unlimited", so the child parsed where it
+  should have refused with `pdf_limits_unavailable` (#662).
 - **Dependencies** — `strands-agents-tools` is held below 0.8.10 (`>=0.8.0,<0.8.10`): the
   exec bridge refuses a release it was not audited against (0.8.9), so a fresh install
   got 0.8.10 and no exec-class tools. Deliberate until a newer release is audited (#296).
+- **Authority and grant stores** — a delete + recreate of the epoch store, latch store,
+  lifecycle log, outbox, watchdog record or grant store is now detected on Linux too: the
+  file identity is device, inode, size and `st_ctime_ns` (recorded by the writer after its
+  own write and rename), not device and inode alone, which ext4 and tmpfs reuse at once
+  (#661). An outside append to the lifecycle log is now caught at the next append
+  (`lifecycle_log_replaced`), not only at the watchdog's next pass.
 
 - **Schedules** — Resume on `/schedules` with the default browser confirmation channel opens the
   confirmation (checkbox) dialog after the resume dialog instead of answering "Cancelled; nothing

conftest.py

@@ -26,6 +26,26 @@
 finally:
     del os.environ["STIRA_DATA_DIR"]
 
+# --- Test HTTP helpers: no reverse DNS at bind (#662) ---------------------------------
+# ``http.server.HTTPServer.server_bind`` asks ``socket.getfqdn(host)``. On a CI macOS
+# runner that reverse lookup can take tens of seconds (it blocked the fake helper servers
+# past their 20-30 s timeouts: four smoke failures and "fake model did not start"). Every
+# helper here binds 127.0.0.1 and no test reads ``server_name``, so the name is the
+# address. Product code does not use ``http.server``; subprocess helpers do the same in
+# their own source.
+import http.server as _http_server
+import socketserver as _socketserver
+
+if not getattr(_http_server.HTTPServer.server_bind, "_stira_no_reverse_dns", False):
+
+    def _server_bind(self):
+        _socketserver.TCPServer.server_bind(self)
+        self.server_name, self.server_port = self.server_address[:2]
+
+    _server_bind._stira_no_reverse_dns = True
+    _http_server.HTTPServer.server_bind = _server_bind
+
+
 if not getattr(FlaskClient.open, "_stira_csrf_wrapped", False):
     _original_open = FlaskClient.open
 

docs/authority-service.md

@@ -39,8 +39,14 @@ starts is therefore always `(incarnation, epoch)`, never the epoch alone.
 
 The file is read without following a symlink and without blocking (a FIFO), and the
 opened descriptor is checked again (regular file, this user, mode `0600`, one link, the
-same device and inode as the path). Before every replace the current file is read again
-and must be exactly the one this service wrote last; otherwise nothing is written.
+same device, inode, size and change time as the path). Before every replace the current
+file is read again and must be exactly the one this service wrote last — the same bytes
+and the same device, inode, size and `st_ctime_ns`, recorded after its own write and
+rename — otherwise nothing is written. The change time is part of the identity because
+Linux (ext4, tmpfs) gives a freed inode number to the next new file at once: a delete and
+recreate with the same bytes keeps the inode number but not the change time (#661). The
+same identity is used by the latch store, the lifecycle log, the outbox, the watchdog's
+record and the grant store.
 
 | Reason code | Meaning |
 |---|---|

docs/authority-service_ja.md

@@ -37,9 +37,13 @@ rename → ディレクトリの flush で書く。最新 16 件の記録を持
 `(インカーネーション, エポック)` で、エポックだけでは決めない。
 
 ファイルはシンボリックリンクをたどらず、ブロックせずに（FIFO でも）開き、開いた記述子を
-もう一度確かめる（通常ファイル、この利用者、モード `0600`、リンク 1 本、パスと同じデバイスと
-inode）。置き換える前には毎回いまのファイルを読み直し、このサービスが最後に書いたものと
-完全に同じでなければ何も書かない。
+もう一度確かめる（通常ファイル、この利用者、モード `0600`、リンク 1 本、パスと同じデバイス・
+inode・サイズ・変更時刻）。置き換える前には毎回いまのファイルを読み直し、このサービスが最後に
+書いたものと完全に同じ（同じバイト、そして自分の書込みと rename の後に記録したデバイス・inode・
+サイズ・`st_ctime_ns`）でなければ何も書かない。変更時刻を同一性に含めるのは、Linux（ext4、
+tmpfs）が解放された inode 番号をすぐ次の新しいファイルに渡すため: 同じバイトでの削除と作り直しは
+inode 番号を保っても変更時刻は保たない（#661）。ラッチストア、ライフサイクルログ、outbox、
+ウォッチドッグの記録、グラントストアも同じ同一性を使う。
 
 | 理由コード | 意味 |
 |---|---|

scripts/check_board_svg_browser.mjs

@@ -85,10 +85,12 @@ async function waitFor(fn, ms = 20000, step = 150) {
 
 async function fakeModel() {
   const p = start(PY, [path.join(ROOT, 'tests/fake_bedrock_endpoint.py'), '0'], { cwd: ROOT });
-  let out = '';
+  let out = '', err = '';
   p.stdout.on('data', (d) => (out += d));
+  p.stderr.on('data', (d) => (err += d));
   const url = await waitFor(async () => (out.match(/on (http:\/\/127\.0\.0\.1:\d+)/) || [])[1]);
-  if (!url) throw new Error('fake model did not start');
+  // The reason, not just the fact (#662: a bind that waited for reverse DNS looked the same).
+  if (!url) throw new Error(`fake model did not start (exit ${p.exitCode}): ${out.slice(-400)}${err.slice(-800)}`);
   return url;
 }
 
@@ -300,7 +302,17 @@ async function run(label, boardSvg) {
     report.requests = { total: reqs.length, api: reqs.filter((u) => u.startsWith(`${BASE}/api/`)).length, data: reqs.filter((u) => u.startsWith('data:')).length, other: foreign };
     check('no request to any probe URL (whole session)', probes.length === 0, probes);
     check('no request other than the page APIs while the reply was shown', foreign.length === 0, foreign);
-    const errors = after.filter((e) => e.method === 'Runtime.exceptionThrown' || (e.method === 'Runtime.consoleAPICalled' && e.params.type === 'error') || (e.method === 'Log.entryAdded' && e.params.entry.level === 'error'));
+    // A CSP report is the policy doing its work, not a page error: Chrome logs a security
+    // entry at error level for the hostile SVG's own inline onload ("Executing inline event
+    // handler violates the following Content Security Policy directive ..." in 156, "Refused
+    // to execute ..." before), which is exactly what must happen. Those are taken out of the
+    // error check and checked on their own: every security entry of this window must be such
+    // a CSP report, and the canary / probe checks above already prove nothing ran (#662).
+    const cspReport = (e) => e.method === 'Log.entryAdded' && e.params.entry.source === 'security' && /Content Security Policy/.test(e.params.entry.text || '');
+    const security = after.filter((e) => e.method === 'Log.entryAdded' && e.params.entry.source === 'security');
+    report.cspReports = security.filter(cspReport).map((e) => (e.params.entry.text || '').slice(0, 200));
+    check('every security log entry is a CSP report (the policy working)', security.every(cspReport), security.map((e) => (e.params.entry.text || '').slice(0, 200)));
+    const errors = after.filter((e) => !cspReport(e) && (e.method === 'Runtime.exceptionThrown' || (e.method === 'Runtime.consoleAPICalled' && e.params.type === 'error') || (e.method === 'Log.entryAdded' && e.params.entry.level === 'error')));
     report.consoleErrors = errors.map((e) => JSON.stringify(e.params).slice(0, 300));
     check('no console error / exception while the reply was shown', errors.length === 0, report.consoleErrors);
     check('no dialog opened', !events.some((e) => e.method === 'Page.javascriptDialogOpening'));

src/stiracore/authority/epoch_store.py

@@ -56,7 +56,10 @@
     "EPOCH_FILE",
     "KEEP_RECORDS",
     "EpochRecord",
+    "Identity",
     "LoadResult",
+    "identity",
+    "install",
     "load",
     "next_record",
     "write",
@@ -116,7 +119,7 @@ class LoadResult:
     status: str  # missing | ok | bad
     records: tuple[EpochRecord, ...] = ()
     reason: str | None = None
-    identity: tuple[int, int] | None = None  # (st_dev, st_ino) of the file that was read
+    identity: Identity | None = None  # :func:`identity` of the file that was read
 
     @property
     def last(self) -> EpochRecord | None:
@@ -212,6 +215,46 @@ def _parse(raw: bytes) -> LoadResult:
     return LoadResult("ok", tuple(records))
 
 
+Identity = tuple[int, int, int, int]
+
+
+def identity(info: os.stat_result) -> Identity:
+    """``(st_dev, st_ino, st_size, st_ctime_ns)``: the file a writer left behind (#661).
+    The device / inode pair alone is not enough on Linux: ext4 and tmpfs hand a freed inode
+    number to the next new file at once, so a delete + recreate (any bytes) between two
+    passes would keep it. Every write, append, rename or chmod sets ``st_ctime``; a writer
+    records this value after its own last change (write, flush, rename), so only a change
+    made by someone else differs. (``grants/progress.py`` uses the same idea.)"""
+    return (info.st_dev, info.st_ino, info.st_size, info.st_ctime_ns)
+
+
+def install(fd: int, tmp: Path, path: Path) -> Identity:
+    """Rename the written temporary file ``tmp`` (open as ``fd``, closed here) over
+    ``path``; the :func:`identity` of the installed file, taken *after* the rename (a
+    rename sets ``st_ctime`` on Linux). On POSIX the descriptor stays open across the
+    rename, so the inode cannot be freed and reused before it is identified; Windows
+    cannot rename an open file, so there the path is checked to still be the written
+    file (same device / file index) right after the rename."""
+    if os.name == "posix":
+        try:
+            os.replace(tmp, path)
+            return identity(os.fstat(fd))
+        finally:
+            os.close(fd)
+    try:
+        written = os.fstat(fd)
+    finally:
+        os.close(fd)
+    os.replace(tmp, path)
+    after = os.lstat(path)
+    if not stat.S_ISREG(after.st_mode) or (after.st_dev, after.st_ino) != (
+        written.st_dev,
+        written.st_ino,
+    ):
+        raise OSError(errno.EEXIST, "replaced right after the rename", str(path))
+    return identity(after)
+
+
 def _unsafe(info: os.stat_result) -> bool:
     """Not a regular, singly linked file of this user with mode ``0600`` or tighter."""
     if not stat.S_ISREG(info.st_mode) or info.st_nlink != 1:
@@ -249,7 +292,7 @@ def load(data_dir: Path) -> LoadResult:
             opened = os.fstat(fd)
         except OSError:
             return LoadResult("bad", reason="epoch_store_unreadable")
-        if _unsafe(opened) or (opened.st_dev, opened.st_ino) != (info.st_dev, info.st_ino):
+        if _unsafe(opened) or identity(opened) != identity(info):
             return LoadResult("bad", reason="epoch_store_unsafe")
         if opened.st_size > _MAX_BYTES:
             return LoadResult("bad", reason="epoch_store_malformed")
@@ -262,7 +305,7 @@ def load(data_dir: Path) -> LoadResult:
     result = _parse(raw)
     if result.status != "ok":
         return result
-    return LoadResult("ok", result.records, identity=(opened.st_dev, opened.st_ino))
+    return LoadResult("ok", result.records, identity=identity(opened))
 
 
 def _flush(fd: int) -> None:
@@ -278,9 +321,9 @@ def _flush(fd: int) -> None:
     os.fsync(fd)
 
 
-def write(data_dir: Path, records: tuple[EpochRecord, ...]) -> tuple[int, int]:
+def write(data_dir: Path, records: tuple[EpochRecord, ...]) -> Identity:
     """Replace the file with ``records`` (the newest :data:`KEEP_RECORDS`), durably; the
-    new file's ``(st_dev, st_ino)``. The caller checks the current file first
+    new file's :func:`identity`. The caller checks the current file first
     (:func:`load`): this function replaces whatever is there."""
     directory = Path(data_dir)
     path = directory / EPOCH_FILE
@@ -297,13 +340,13 @@ def write(data_dir: Path, records: tuple[EpochRecord, ...]) -> tuple[int, int]:
             while view:
                 view = view[os.write(fd, view) :]
             _flush(fd)
-            written = os.fstat(fd)
-        finally:
+        except BaseException:
             os.close(fd)
-        os.replace(tmp, path)
+            raise
+        written = install(fd, tmp, path)
     except BaseException:
         with contextlib.suppress(OSError):
             os.unlink(tmp)
         raise
     fsync_dir(directory, _flush)
-    return (written.st_dev, written.st_ino)
+    return written

src/stiracore/authority/latches.py

@@ -217,7 +217,7 @@ def __init__(self, data_dir: Path | None) -> None:
         self._dir = None if data_dir is None else Path(data_dir)
         self._lock = threading.Lock()
         self._records: list[dict] = []
-        self._identity: tuple[int, int] | None = None
+        self._identity: tuple[int, ...] | None = None
         self._bad: str | None = None
         if self._dir is None:
             return

src/stiracore/authority/lifecycle_events.py

@@ -34,11 +34,12 @@
 import math
 import os
 import re
+import stat
 import threading
 from pathlib import Path
 from typing import Any
 
-from . import private_file
+from . import epoch_store, private_file
 from .latches import LATCHES
 
 __all__ = [
@@ -494,8 +495,8 @@ def __init__(self, data_dir: Path | None, *, rotate: bool = True) -> None:
         self._dir = None if data_dir is None else Path(data_dir)
         self._lock = threading.Lock()
         self._bad: str | None = None
-        self._identity: tuple[int, int] | None = None
-        self._old_identity: tuple[int, int] | None = None
+        self._identity: tuple[int, ...] | None = None
+        self._old_identity: tuple[int, ...] | None = None
         self._size = 0
         self._last_hash = _ZERO
         self._last_seq = 0
@@ -575,7 +576,16 @@ def _rotate_locked(self) -> None:
         from stiracore.durable_dir import fsync_dir
 
         fsync_dir(self._dir)
-        self._old_identity = self._identity
+        # A rename sets st_ctime (Linux), so the sealed segment's identity is taken again;
+        # it must still be the file this log appended to last (#661).
+        moved = os.lstat(self._dir / OLD_LOG_FILE)
+        if (
+            self._identity is None
+            or not stat.S_ISREG(moved.st_mode)
+            or (moved.st_dev, moved.st_ino, moved.st_size) != self._identity[:3]
+        ):
+            raise private_file.Replaced("replaced")
+        self._old_identity = epoch_store.identity(moved)
         self._old_digest = self._cur_hash.hexdigest()
         self._old_size = self._size
         self._identity = None

src/stiracore/authority/outbox.py

@@ -245,7 +245,7 @@ def __init__(
         self._items: list[dict] = []
         self._seen: set[str] = set()
         self._acked: set[str] = set()
-        self._outbox_identity: tuple[int, int] | None = None
+        self._outbox_identity: tuple[int, ...] | None = None
         self._outbox_seen: bytes | None = None
         if self.log.unhealthy_reason is not None:
             self._bad = self.log.unhealthy_reason