Title
Independent empirical audit of OmegaClaw-Core: documentation-vs-engine discrepancies, provenance blindness, and belief poisoning under revision (all reproducible)
Body
Hi maintainers — first, thank you for open-sourcing OmegaClaw-Core; the design intent around auditable inference and transparent implementation made it an ideal target for an independent empirical audit. I ran a fully contained, reproducible audit against the real engine (PyMeTA hyperon==0.2.10) over two days and found several concrete discrepancies between the documentation and live behavior. Everything is reproducible in seconds; all artifacts are in the linked repository.
Documentation vs engine (measured, not read):
- Contradiction example: docs
(0.395, 0.875) — engine (stv 0.34 0.833)
- NAL analogy asymmetry: docs
0.36 — engine (stv 0.25 0.09) (argument-order effects)
- PLN abduction orientation: docs
(robin bird) (0.767, 0.422) — engine (Inheritance bird robin) (1.0, 0.4475)
- Confidence decay (4-hop): docs quote
<0.5 (hop 3) / ~0.25 (hop 4) — engine 0.81 → 0.729 → 0.656 → 0.590 (premise-dependent)
Adversarial findings (31-case battery, 24 PASS / 3 DISCREPANCY / 4 INFO):
- Premise swap is silently accepted (8/8) — term-order swaps are invisible to the reasoner; no polarity check in truth propagation
- Provenance blindness — a confident lie and a verified claim get identical numeric treatment from the truth function
- Revision absorbs poison — four revisions from a
0.6/0.5 counter-source drift truth 0.915 → 0.842 while confidence grows 0.909 → 0.929
- Documented counter-example — a persistent AtomSpace does accumulate and revise across turns, contradicting the "fresh AtomSpace per
(metta |-) call" claim
What validated well: deterministic NAL/PLN truth calculus, reversible self-modification (18→19→18 with proof invariants), auth-gated channel access (real channels/irc.py against a local server, 13/13 scenarios), and the triage/novelty-modulation extension surface.
Suggestions (open to maintainers' judgment):
- Re-run the tutorial numeric examples and update the docs (or pin the interpreter version they target)
- Consider a polarity/provenance check in truth propagation — the 8/8 swap acceptance is the single highest-impact finding
- Consider documenting revision's confidence-growth-under-poison property, and whether a triage gate (like the pure-MeTTa
lib_triage.metta in the audit) should be a default layer
Reproduce everything (~20 s on a 2-core sandbox):
python research/final_metrics.py
python research/harness.py
PYTHONPATH=$PWD python research/irc_integration.py --mode all
Full audit: https://github.com/NullLabTests/OmegaClaw — CAMPAIGN_SUMMARY.md, RESEARCH_LOG.md, research/ARCHITECTURE_FINAL.md, and every raw measurement under research/measurements/.
Happy to open this as separate issues per finding, or turn any of it into a PR (e.g., doc fixes) if useful.
Title
Independent empirical audit of OmegaClaw-Core: documentation-vs-engine discrepancies, provenance blindness, and belief poisoning under revision (all reproducible)
Body
Hi maintainers — first, thank you for open-sourcing OmegaClaw-Core; the design intent around auditable inference and transparent implementation made it an ideal target for an independent empirical audit. I ran a fully contained, reproducible audit against the real engine (PyMeTA
hyperon==0.2.10) over two days and found several concrete discrepancies between the documentation and live behavior. Everything is reproducible in seconds; all artifacts are in the linked repository.Documentation vs engine (measured, not read):
(0.395, 0.875)— engine(stv 0.34 0.833)0.36— engine(stv 0.25 0.09)(argument-order effects)(robin bird) (0.767, 0.422)— engine(Inheritance bird robin) (1.0, 0.4475)<0.5(hop 3) /~0.25(hop 4) — engine0.81 → 0.729 → 0.656 → 0.590(premise-dependent)Adversarial findings (31-case battery, 24 PASS / 3 DISCREPANCY / 4 INFO):
0.6/0.5counter-source drift truth0.915 → 0.842while confidence grows0.909 → 0.929(metta |-)call" claimWhat validated well: deterministic NAL/PLN truth calculus, reversible self-modification (18→19→18 with proof invariants), auth-gated channel access (real
channels/irc.pyagainst a local server, 13/13 scenarios), and the triage/novelty-modulation extension surface.Suggestions (open to maintainers' judgment):
lib_triage.mettain the audit) should be a default layerReproduce everything (~20 s on a 2-core sandbox):
python research/final_metrics.py python research/harness.py PYTHONPATH=$PWD python research/irc_integration.py --mode allFull audit: https://github.com/NullLabTests/OmegaClaw —
CAMPAIGN_SUMMARY.md,RESEARCH_LOG.md,research/ARCHITECTURE_FINAL.md, and every raw measurement underresearch/measurements/.Happy to open this as separate issues per finding, or turn any of it into a PR (e.g., doc fixes) if useful.