A framework-independent behavioural contract for agentic datasets, expressed as language-neutral executable vectors and evaluated against implementations without access to their internals.
15 normative assertions, 85 language-neutral vector steps
reference architecture 4 runtimes x 2 dataset boundaries 15/15 each
independent implementation shares no code with the above 15/15
mutation analysis 17 / 17 targeted violations detected
15 / 15 assertions independently exercised
2.2 detecting assertions per mutant (mean)
not treated as an optimization metric
defects exposed by the
conversion to vectors F-010, F-011
execution safety 0 / 576 prohibited executions
0 / 24 in the evaluation set
tests 405 passed
Authorized Recall@5 filter after truncation 0.853
filter before truncation 0.960
+0.107
The four runtimes are the experimental variable, not the product. What is being tested is whether the contract survives being expressed somewhere else — and whether a harness that cannot see inside an implementation can still tell a conforming one from a broken one.
Reproduce with agentic-dataset-conformance run --subject conformance.subjects:subjects --matrix.
Raw output is in docs/runs/; every number's caveats are in
docs/RESULTS.md.
Public before
v0.1.0is tagged, deliberately. Nothing is archived or minted yet, so a finding can still change the artifact rather than becoming errata against a DOI. If you find something, open an issue — a finding that an assertion is ambiguous is more useful here than a passing run.What this is not. No deployment, no real data, no model in the loop by default, no latency or cost claim, and no security guarantee — see
docs/CLAIMS.md, which is frozen and lists the one thing explicitly not claimed. The independent implementation is a 250-line toy written by the same person who wrote the specification: that establishes independence from the reference code, not from its author's reading of the contract. Interpretive independence is the next threshold and it needs somebody else —CONTRIBUTING.mdsays how.
An agentic dataset is a dataset that describes itself, advertises bounded capabilities, accepts a semantic intent, decides whether an action is admissible, executes only what was admitted, refuses the rest, and leaves evidence.
The model is not new here, and neither are its mechanisms: ok-governed-motion
implements the three-valued verdict in Rust for robot motion, and
ok-nfcore-admission-gate implements the per-task admission gate, measured on
a real bioinformatics workload. Both are public and linked below.
What did not exist was one artifact a third party can run, point at, and disagree with — a contract stated precisely enough to be implemented twice, with a way to check whether two implementations agree. That is what this repository is.
If you have never heard of any of this, start with
CONFORMANCE.md: fifteen assertions, each naming a failure
it rules out. Everything else here exists to check them.
The governance model is not a property of a framework.
Three architecture documents that agree with each other prove nothing — they
were written by one person from one model. So the fifteen assertions in
CONFORMANCE.md are implemented once and run against every
runtime:
RUNTIME RESULT PASSED Where admission routes
native+local PASS 15/15 a function call
langgraph+local PASS 15/15 a conditional edge
llamaindex+local PASS 15/15 typed event dispatch
adk+local PASS 15/15 a graph node + before-tool callback
native+mcp PASS 15/15 (each of the four again, with every
langgraph+mcp PASS 15/15 dataset behind a real MCP session)
llamaindex+mcp PASS 15/15
adk+mcp PASS 15/15
AD-015 prohibited execution rate: 0.000 (target exactly 0)
The eight reference configurations share one ControlPlane, deliberately: an
assertion that passed because each port re-implemented its own policy would be
eight experiments, not one. The ninth subject shares nothing.
packages/agentic-dataset-conformance/src/agentic_dataset_conformance/toy.py is 250 lines written from the specification
— grants are integers in a dict, the cache is a dict, there is no framework, no
MCP and no policy engine — and it passes all fifteen. That is the evidence that
the assertions are properties of the contract rather than of the reference
architecture.
It is still weaker evidence than an implementation written by someone else, and
docs/RESULTS.md §1 says why.
The gap above is the interesting one, and it cannot be closed from inside this
repository. An implementation written by someone else from
CONFORMANCE.md alone, in any language, without using this
reference ControlPlane, would test whether the specification is complete
enough to be implemented twice and whether two implementations agree on the
governance semantics.
The harness can now check one. Implement ConformanceSubject
(interface.py, four methods),
register it in conformance/subjects.py, and run
the same vectors. packages/agentic-dataset-conformance/src/agentic_dataset_conformance/toy.py is a worked example of
exactly that.
A finding that an assertion is ambiguous is a more useful result than a passing
run. docs/PORTABILITY.md records what the contract can
and cannot reach, including the one property that was deliberately widened and
the one that cannot be checked from outside at all.
The suite failed on this implementation five times before it passed, twice only
in the MCP configuration and twice only under the async runtimes. Those are
written down in docs/FINDINGS.md.
The LLM may interpret, propose, rank and explain.
The control plane decides whether execution is allowed.
Admission returns one of three verdicts, and only an approval mints the token that permits execution:
GRANTED -> approval token -> execution reachable
REFUSED -> no token -> execution unreachable
INDETERMINATE -> no token -> execution unreachable
INDETERMINATE is not a refusal. An evaluator that is unreachable or out of
budget has not decided anything, and recording that as a refusal invents an
authority nobody exercised. The two serialised reasons —
EVALUATOR_UNAVAILABLE, EVALUATOR_TIMEOUT — and their rationales are copied
from ok-governed-motion's policy.rs, and
tests/test_verdict_parity.py reads that file to check they have not drifted.
Two pieces are published distributions, independently installable and permissively licensed:
# Neither is on PyPI yet -- 0.1.0rc1 is built and waiting. From a clone:
pip install ./packages/agentic-dataset-conformance # contract + vectors + runner
pip install ./packages/authorized-recall # the metric, no dependencies
agentic-dataset-conformance run # against its own worked example
agentic-dataset-conformance run --matrix # and the 17 broken variants
agentic-dataset-conformance vectors --export ./vectors # CC0, take themNeither needs this repository. The reference implementation is not on PyPI
— it is BUSL-1.1, and pip install reads as open source.
For the reference implementation itself, from a clone:
pip install -e ./packages/authorized-recall -e ./packages/agentic-dataset-conformance -e ".[all]"
agentic-dataset-conformance run --subject conformance.subjects:subjects # portable suite, every subject
agentic-dataset-conformance run --subject conformance.subjects:subjects --matrix # the 17-mutant detection matrix
python -m agentic_dataset.reference_suite # white-box suite, 8 configurations
python conformance/generate.py # regenerate world and vectors
pytest -q # 405 tests
python -m authorized_recall # milestone M6, the metric
python evals/evaluate.py # milestone M5, six evaluators
python -m authorized_recall.corpus # regenerate the corpusThe conformance runner exits non-zero on any failure, so it works as a CI gate. Runtimes whose framework is not installed are reported as skipped rather than quietly omitted — a suite that shrinks silently is a suite that always passes.
A minimal run:
from agentic_dataset.adapters import NativeRuntime
from agentic_dataset.datasets import build_control_plane, principals
from agentic_dataset.runtime import Request
runtime = NativeRuntime(build_control_plane())
result = runtime.run(Request(
text="Compare the recovery of batches B001 and B002",
principal=principals()["process_engineer"],
))
print(result.decision, result.reason, result.result)
# GRANTED PRINCIPAL_AUTHORIZED {'batch_ids': ['B001', 'B002'], ...}
refused = runtime.run(Request(
text="Delete the source records",
principal=principals()["process_engineer"],
dataset="purification-batches", capability="delete_source",
))
print(refused.decision, refused.grant, refused.execution.tool_calls)
# REFUSED None []The second example is the one that matters. The test is not that a refusal message was produced; it is that after a refusal there was no capability to execute with.
| Layer | Choice | Why |
|---|---|---|
| Core control plane | No dependencies | If the governance model needed a framework, the claim above would be false. Everything in src/agentic_dataset/ outside adapters/ and mcp_boundary.py is standard library |
| Orchestration | LangGraph · LlamaIndex Workflows · Google ADK | Three mainstream runtimes with different primitives. Admission must be a routed edge, a typed event or a callback — never a prompt |
| Dataset boundary | MCP | A dataset exposes descriptor, schema, lineage and policy as resources and its capabilities as tools; new datasets become discoverable without rewiring anything |
| Policy | Deterministic evaluator, in code | Authority is not a probabilistic judgement. Swapping in Cedar or OPA changes one class and no assertion |
| Evidence | Hash-chained append-only ledger, separate from runtime state | Orchestration checkpoints and an audit record have different lifetimes and different readers |
src/agentic_dataset/
verdict.py the three-valued verdict, ported from ok-governed-motion
descriptor.py dataset contract: schemas, capabilities, prohibitions, provenance
principal.py principals and authorization scopes
intent.py natural language -> structured intent (rule-based or LLM)
admission.py deterministic policy evaluation -> Verdict
grant.py the approval token: minting, expiry, HMAC verification
capabilities.py bounded operations; the wrapper nothing gets past
cache.py authorization-scoped semantic cache
discovery.py policy-aware discovery and Authorized Recall@K
delegation.py the MCP and A2A seams (AD-013, AD-014)
provenance.py evidence records
ledger.py hash-chained append-only ledger
runtime.py the control plane: nodes, state, RunResult
mcp_boundary.py a dataset behind MCP, and the client that consumes it
adapters/ native · langgraph · llamaindex · adk
conformance/ the portable harness: interface + runner, no impl imports
reference_suite/ the white-box suite, which needs implementation access
datasets/ the synthetic reference dataset family
packages/
authorized-recall/ the metric as its own Apache-2.0 distribution
conformance/ the normative artifact: world, vectors, verbs,
an independent implementation and 17 broken variants
examples/ one runnable script per runtime, plus the MCP boundary
tests/ 405 tests
evals/ the M5 evaluators and the committed corpus record
docs/ architecture (three ports), results, findings, raw runs
Browsable at agenticdatasets.org/reference — the same files, with search and a stable anchor per assertion. The site has no content of its own; CI fails if it drifts from the repository.
CONFORMANCE.md— AD-001 … AD-015, the fifteen assertions any implementation must satisfy in any framework. This is the specification; everything else in the repository is one worked example of it.packages/authorized-recall/README.md— Authorized Recall@K: the definition, the two conventions, and the proof that the pre/post-filter gap is non-negative. A separate distribution under Apache-2.0 with no dependency on the control plane, so the metric can be used — and installed — without adopting any of this.docs/CLAIMS.md— everything this project asserts, and the one thing it explicitly does not. Frozen before release.docs/PORTABILITY.md— what the portable contract reaches, what it deliberately does not, and the mutation results.CONTRIBUTING.md— how to write an independent implementation and register it as a subject.RELEASE.md— what has to change together before this is published.docs/RESULTS.md— what was measured, with the caveats attached to each number.docs/FINDINGS.md— where the implementation disagreed with the architecture, and the six defects the suite found.docs/ARCHITECTURE.md— LangChain / LangGraph / MCP / LangSmith. Design, written before the code.docs/ARCHITECTURE-LLAMAINDEX.md— LlamaIndex Workflows variant.docs/ARCHITECTURE-ADK.md— Google ADK variant.PLAN.md— milestones M1–M6 and the open questions, with the answers that were taken.
The three architecture documents are design, and predate the
implementation. Where the code disagrees with them, docs/FINDINGS.md says
so and says why; they have not been retrofitted to match.
Not one license, and not open source as a whole. The split is arranged so that everything needed to implement and test the contract independently is openly licensed, while the reference implementation is not:
| What | License |
|---|---|
| Specification and normative prose | CC BY 4.0 |
| Normative worlds and vectors | CC0-1.0 — vendor them unchanged, no attribution required |
| Conformance software, generator, toy subject, mutants | Apache-2.0 |
| Authorized Recall | Apache-2.0 |
| Reference implementation | BUSL-1.1, converting to Apache-2.0 on 2029-09-02 |
The Agentic Dataset specification, normative vectors, conformance tooling and Authorized Recall implementation are openly licensed for independent implementation and reuse. The reference implementation is source-available under the Business Source License 1.1.
Writing an independent implementation against CONFORMANCE.md and the
vectors, in any language, for any purpose including a commercial one, requires
no permission from anybody. LICENSE.md is the map, with the
per-tier reasoning and the BSL parameters.
The model is argued in three conference papers, all accepted for 2026 and none yet in the published proceedings, so there are no DOIs to cite yet:
| Venue | Title |
|---|---|
| IEEE CCECE 2026 | Agentic Datasets as an Engineering Control Plane |
| IEEE EMBC 2026 | Dataset Descriptors for Autonomous and Observable Biomedical Data Pipelines |
| IEEE BigDataService 2026 | Agentic Data Services: A Control-Plane Architecture for Adaptive Data Workflows |
Nothing in this repository depends on those papers. The assertions, the vectors and the measurements stand on their own and are reproducible from a clone. The papers are where the model is motivated; this is where it is checked.
ok-governed-motion— the three-valued verdict, in Rust, for robot motion rather than datasets. IEEE CBS 2026.tests/test_verdict_parity.pychecks this port against its serialised strings.ok-nfcore-admission-gate— the same admission idea as a per-task gate, measured on a real nf-core workload.agentic-datasets/programme— the programme-level description of what agentic datasets are. CC BY 4.0.
The papers above are the citable statements of the model. This repository is the runnable one.