Skip to content

docs(research): five assessments — NVIDIA/OpenShell/OpenWorker, Agent Substrate, AutoBenchmark, DeepSeek Harness (corrected), Context Language Models - #256

Merged
praxagent merged 11 commits into
mainfrom
docs/context-language-models
Oct 3, 2026
Merged

praxagent merged 11 commits into
mainfrom
docs/context-language-models

Conversation

@praxagent

@praxagent praxagent commented Oct 1, 2026 •

Copy link
Copy Markdown
Owner

This PR now carries all five open research assessments. #246, #251, #252 and #253 were a stacked chain, and stacked PRs re-conflict after every squash merge. They are folded in here and closed with a pointer. Merge this one.


Original #256 description

Assessment of Context Language Models (UW / Meta; code).

Stacked on #253, the end of the research-doc chain.

Verdict: document + adopt the mechanism, rebuilt under Prax's invariants.

  • The method. The model compacts its own context by editing a file mirror of the conversation. Around that: budget nudges, rollback-then-ask on overflow, and free edit-only turns.
  • The results. Zero-shot it beats MEM1, Self-Compact, ACM, RLM and Codex-style summarisation, e.g. BrowseComp-Plus +11.4% at 21.5% fewer FLOPs.
  • Why it moves up the queue. It's the third sighting after ACM and OptMem, so the queued context_compact/context_recall row moves up.
  • Unsafe as published. Only the system prompt and task are protected, roles are rewritable, and tool results fold into user. Injected text could be laundered into the user's voice. The authors flag this risk but don't evaluate it.
  • Prax's version:
    • pins every user turn and approval;
    • keeps roles and provenance tags immutable;
    • never edits the record (a summary plus a pointer, and context_recall).
  • Bank: evolved compaction instructions as a chore(main): release 0.15.0 #29 target (+35.9 held-out), and per-call context snapshots.
  • Not adopted: RL and Suffix Cache Reuse (GPU wall), and unrestricted bash-file editing.
  • Caveats: the code is CC BY-NC, so nothing is vendored. EdgeBench is reported as best of three seeds. I read the code and paper, and ran nothing.

Document + adopt two ideas; confirmation of the Muse-parity containment.
- An out-of-band wire record in the secrets proxy — the tool calls the model
  returned, hashed, append-only, where Prax can't write — checked against
  Prax's own trace, so a compromised Prax can't hide activity by editing it.
- A diff of newly granted access for egress-policy changes and timed grants.
The DPU hardware is out of reach; OpenShell (0.1.x) is a peer to watch.
OpenWorker (Andrew Ng et al., MIT): the closest peer to Prax's governance
stance. Adopt hard floors — a declared set enforced after every rule that can
lower risk; Prax's earned trust can lower two login steps from HIGH to MEDIUM
on self-reported success today — and parked approvals for unattended runs
instead of refusing and losing the work. Plus approval provenance per call.

OpenShell's product page adds per-program network policy and a policy prover
with an access ceiling: queue the ceiling, and a time-boxed evaluation of
OpenShell as prax-sandbox's runtime.
Not ruled out: Prax should be highly competitive with OpenShell. Candidate
routes recorded — per-program proxy identity inside the sandbox, cgroup/eBPF
attribution, or OpenShell's supervisor after the evaluation.
…nnel-held identity

Google's CNCF sandbox application (cncf/sandbox#523). Its egress design is the
closest published match to the secrets proxy. Adopted: never inject into
cleartext (ours did; fixed in prax-secrets-proxy #7). Queued: a trusted tunnel
client holds the sandbox's proxy identity, so the program can't read it. The
platform is a scale non-goal; its DNS bypass matches our documented gap.
…or-all means audit the key

Meta RAM's agent-built benchmarks saturate unaided; detailed human specs halve
solver scores. Adopt: difficulty of LLM-authored cases measured on a solver
from another provider, and a case every solver fails gets its answer key
audited — lowest-score selection also selects wrong keys.
dsh sandboxes report full/partial enforcement; Prax's hand-installed
containment is never checked from inside. Bank: scrubbed child environments
(69 subprocess calls, none sets env=; exposure unverified) and an advisory
reminder before the hard turn limits. Monotonic deny-only guards confirm
hard floors.
…ing it, and correct it

The September README-only note said dsh has no governance layer. The code
has approvals, deny-only guards and a process sandbox; what it lacks is
network policy and audit. One page, one index entry, the correction stated.
… under Prax's invariants

UW/Meta CLMs edit a file mirror of their own context and beat ACM, RLM and
summarisation zero-shot. Third sighting with ACM and OptMem, so the queued
context_compact/context_recall row moves up — but rebuilt: user turns and
approvals pinned, roles and provenance immutable, the record never edited.
As published, roles are rewritable and tool results fold into user. Code is
CC BY-NC; nothing vendored.
Entries added since were inserted between the Muse entry and its indented
continuation, so its Sentinel/authd/verdict bullets rendered under the wrong
entry. Pure move; no line changed.
@praxagent
praxagent force-pushed the docs/context-language-models branch from 2915144 to 9f69a38 Compare October 2, 2026 05:34
@praxagent praxagent changed the title docs(research): Context Language Models — adopt agent-managed context under Prax's invariants docs(research): five assessments — NVIDIA/OpenShell/OpenWorker, Agent Substrate, AutoBenchmark, DeepSeek Harness (corrected), Context Language Models Oct 2, 2026
@praxagent
praxagent merged commit f316c30 into main Oct 3, 2026
1 check passed
@praxagent
praxagent deleted the docs/context-language-models branch October 3, 2026 01:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant