Skip to content

[OMEGA-294] Independent empirical audit: doc-vs-engine discrepancies, provenance blindness, belief poisoning under revision (all reproducible) #302

Description

@NullLabTests

Title

Independent empirical audit of OmegaClaw-Core: documentation-vs-engine discrepancies, provenance blindness, and belief poisoning under revision (all reproducible)

Body

Hi maintainers — first, thank you for open-sourcing OmegaClaw-Core; the design intent around auditable inference and transparent implementation made it an ideal target for an independent empirical audit. I ran a fully contained, reproducible audit against the real engine (PyMeTA hyperon==0.2.10) over two days and found several concrete discrepancies between the documentation and live behavior. Everything is reproducible in seconds; all artifacts are in the linked repository.

Documentation vs engine (measured, not read):

  • Contradiction example: docs (0.395, 0.875) — engine (stv 0.34 0.833)
  • NAL analogy asymmetry: docs 0.36 — engine (stv 0.25 0.09) (argument-order effects)
  • PLN abduction orientation: docs (robin bird) (0.767, 0.422) — engine (Inheritance bird robin) (1.0, 0.4475)
  • Confidence decay (4-hop): docs quote <0.5 (hop 3) / ~0.25 (hop 4) — engine 0.81 → 0.729 → 0.656 → 0.590 (premise-dependent)

Adversarial findings (31-case battery, 24 PASS / 3 DISCREPANCY / 4 INFO):

  • Premise swap is silently accepted (8/8) — term-order swaps are invisible to the reasoner; no polarity check in truth propagation
  • Provenance blindness — a confident lie and a verified claim get identical numeric treatment from the truth function
  • Revision absorbs poison — four revisions from a 0.6/0.5 counter-source drift truth 0.915 → 0.842 while confidence grows 0.909 → 0.929
  • Documented counter-example — a persistent AtomSpace does accumulate and revise across turns, contradicting the "fresh AtomSpace per (metta |-) call" claim

What validated well: deterministic NAL/PLN truth calculus, reversible self-modification (18→19→18 with proof invariants), auth-gated channel access (real channels/irc.py against a local server, 13/13 scenarios), and the triage/novelty-modulation extension surface.

Suggestions (open to maintainers' judgment):

  1. Re-run the tutorial numeric examples and update the docs (or pin the interpreter version they target)
  2. Consider a polarity/provenance check in truth propagation — the 8/8 swap acceptance is the single highest-impact finding
  3. Consider documenting revision's confidence-growth-under-poison property, and whether a triage gate (like the pure-MeTTa lib_triage.metta in the audit) should be a default layer

Reproduce everything (~20 s on a 2-core sandbox):

python research/final_metrics.py
python research/harness.py
PYTHONPATH=$PWD python research/irc_integration.py --mode all

Full audit: https://github.com/NullLabTests/OmegaClawCAMPAIGN_SUMMARY.md, RESEARCH_LOG.md, research/ARCHITECTURE_FINAL.md, and every raw measurement under research/measurements/.

Happy to open this as separate issues per finding, or turn any of it into a PR (e.g., doc fixes) if useful.


Metadata

Metadata

Assignees

No one assigned

    Labels

    in-jiraThe issue has been accepted for fixing

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions