feat: add relyable scanner adapter (functional re-derivation evidence)

[maxgris]

What

Adds relyable as a registry-backed scanner adapter — deliberately not
preinstalled in the runtime image (see Execution safety) — emitting graded
functional re-derivation evidence for a skill — does it still reproduce the author's own committed spec?

clawscan ./skills/csv-summarizer --scanner relyable

The adapter shells out to relyable-scan <target> --json
(veriker/relyable, Python ≥3.11; installed on the
host via clawscan install relyable, or into a derived sandbox image for Docker mode)
and preserves its
relyable-scan-v1 payload as raw evidence, exactly like the other command-backed
scanners.

Why this belongs in a security-scan harness

It's a different, complementary axis — and it's deliberately labeled as such
("axis": "functional-rederivation" is stamped in-schema). Provenance proves
authorship; the security scanners prove it won't hurt you; neither proves the skill
does what its docs claim. A skill whose behavior has drifted from its own documented
contract — regression, post-publish tampering, or an honest bug — is a trust signal the
current scanner set can't produce. Relyable emits the strongest grade that applies:

• exogenous — a declared rederive.json property manifest (idempotence /
  round-trip); relyable computes both sides of the relation from the skill's own code
  and mutation-tests the result against vacuity, so the author never votes on the
  expected value
• self_spec — the author's own committed oracle (shipped test suite > documented
  I/O examples in SKILL.md/README > paired example fixtures), re-run deterministically
  with per-tool REPRODUCES / CONTRADICTS / UNJUDGEABLE_*
• cold_golden — when an LLM key is set, a code-blind model infers goldens from
  SKILL.md alone (it never sees source) and a mechanical compare grades them;
  it abstains unless the docs pin exact behavior, passes are mutation-tested, and a
  divergence is reported as unconfirmed, never as an accusation
• non_rederivable — nothing checkable: an honest hole, never a fabricated pass

Evidence, not policy, in line with the harness's evidence-first invariant. It does
not detect malware or prompt injection and never claims to.

Measured on your public corpus (clawhub-security-signals, eval_holdout, n=3368):
0 skills declare a re-derivation manifest, 15 (0.45%) ship any committed oracle, and
343 (10.2%) have a runnable entrypoint. Running the cold lane over all 343 runnable
skills: 330 abstain (327 because the docs do not pin exact output — the integrity
rules refuse to guess — plus 3 unparseable constructor responses), 2 pass
non-vacuously (mutation kill 25% and 8%), 5 reproduce but are demoted as vacuous,
3 diverge (reported unconfirmed), 3 unjudgeable (every constructed golden errored).
We read that distribution as
the point, not a limitation: the ecosystem's functional-trust bottleneck is
author-committed oracles, and the scanner reports honest floors instead of inflating
coverage. One documented I/O example moves a skill to self_spec; a one-file
manifest moves it to exogenous.

Execution safety

Grading executes skill code, so the scanner is fail-closed: without an explicit
disposable-host acknowledgement it runs nothing — the self_spec lane reports
UNJUDGEABLE_NO_SANDBOX per tool, and the exogenous/cold lanes record an explicit
no-ack degrade reason. The adapter passes --allow-host-exec only in Docker sandbox
mode; with --sandbox off, operators opt in via RELYABLE_SCAN_ALLOW_HOST_EXEC=1.
Skill code that does run gets a scrubbed environment allowlist, so an env-dumping
skill cannot exfiltrate a forwarded LLM key into the evidence. Key forwarding itself
is narrowed to an explicit opt-in: only RELYABLE_-prefixed vars ride the sandbox
env passthrough, never an operator's generic provider keys. No required env vars;
no secret-bearing flags; the payload records LLM-key availability as a presence
boolean only.

Runtime image: unchanged. Per the maintainer's preference to keep the runtime
image lean, the net PR diff leaves docker/clawscan-runtime/Dockerfile byte-identical
to main — relyable is registry-backed but not preinstalled. Operators opt in explicitly:
clawscan install relyable for host mode, or a two-line derived image
(recipe in docs/scanners.md#relyable-and-the-sandbox) passed via --sandbox-image
or a profile's sandbox.image to keep Docker-mode execution. Against the stock
image, selecting relyable records an explicit failed scanner result whose error
points at those two paths, and nothing executes.

Changes

• internal/runner/relyable_scanner.go — the adapter (snyk-shaped: JSON-on-stdout
  wins; non-zero exit with valid JSON → completed + error recorded; empty/invalid →
  failed). An executable file not found failure points at the derived-image /
  clawscan install paths, since the tool is not preinstalled
• internal/runner/scanner_registry.go — registry entry (command-backed, installable;
  the install plan mirrors the derived-image recipe, including --with pytest, so a
  host install can re-run shipped test suites too).
  The cold lane's LLM access is an explicit per-scanner opt-in: only RELYABLE_-prefixed
  env vars are listed for sandbox forwarding, and generic
  ANTHROPIC_API_KEY/OPENAI_API_KEY are deliberately NOT auto-forwarded (standalone
  relyable-scan honors them; ClawScan never hands them over implicitly). Defense in
  depth behind that: relyable executes skill code under a scrubbed environment
  allowlist, so even a forwarded key can't be exfiltrated by an env-dumping skill into
  the evidence.
• internal/runner/relyable_scanner_test.go — 6 tests incl. the sandbox/ack matrix and the missing-executable hint
• scanner_registry_test.go, cmd/clawscan/main_test.go — roster fixtures updated
• docs/scanners.md + README.md — the catalog row (both copies of the Available
  scanners table, per the socket-adapter precedent) plus a "Relyable and the sandbox"
  section: clawscan install relyable for host mode; the derived-image recipe for
  Docker mode (--with pytest because relyable re-runs an author's shipped test suite
  via its own interpreter, and uv tool venvs are isolated)
  go test ./... green; gofmt -l clean.

Example evidence (excerpt of a real sandbox-mode run, drifted skill)

Produced by this patch's adapter in --sandbox docker, with relyable installed at a
pinned ref into a derived image built FROM ghcr.io/openclaw/clawscan-runtime:latest
per the docs recipe and selected via --sandbox-image — the sandbox is the
disposable-host ack, so the adapter passes --allow-host-exec and the drifted code
actually executes:

{
  "schemaVersion": "relyable-scan-v1",
  "axis": "functional-rederivation",
  "skills": [{
    "skill": "drifted-adder",
    "grade": "self_spec",
    "selfSpecTier": "S-B",
    "verdict": "DIVERGED",
    "tools": {"drifted-adder:add": "CONTRADICTS"},
    "degradeReasons": {"exogenous": "no rederive manifest (spec-ref or property) declared"}
  }],
  "honesty": {
    "unfakeableClaimReservedFor": "exogenous",
    "selfSpecMeaning": "consistency with the author's own committed spec, not correctness",
    "securityAxis": "not covered: ... run alongside security scanners"
  }
}

On the README's own demo target (Trail of Bits csv-summarizer @ 4ffbf946) — runs
performed on this PR's earlier heads; the grading path is unchanged by this revision —
relyable reports non_rederivable / OUT_OF_SCOPE: the skill commits no checkable oracle (no
tests, no documented I/O examples), and in Docker sandbox mode with an LLM key present
the cold constructor ABSTAINS — the docs pin no output format, so the integrity rules
refuse to guess and no code is executed for grading. Without the sandbox ack, the
scanner fail-closes to the same honest floor with an explicit no-ack degrade reason
recorded per lane.

Follow-ups we'd propose separately (not in this PR)

• A benchmark profile including relyable, submitted through the documented
  proposals/ → SkillTrustBench leaderboard path, so the evidence value is judged on
  numbers.
• Extending the rederive.json manifest convention (spec-refs and more property
  kinds beyond idempotence/round-trip) — the corpus's zero-manifests count above is
  the demand case, and it dovetails with the capability-manifest direction.

🤖 Generated with Claude Code

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed July 17, 2026, 10:37 PM ET / July 18, 2026, 02:37 UTC.

Summary
The PR adds a built-in relyable command-backed scanner, Docker runtime installation, registry/help tests, and scanner catalog documentation.

Reproducibility: not applicable. as a bug report: this is a new scanner integration, and the PR body supplies a concrete claimed Docker-mode run rather than a failing current-main reproduction.

Review metrics: 2 noteworthy metrics.

• Patch surface: 8 files affected; 265 added, 3 removed. The change spans adapter behavior, the built-in registry, runtime image, tests, and two public scanner catalogs.
• Runtime dependencies: 1 new Git-revision tool install plus pytest. The Docker image change is the key non-test execution and supply-chain surface for maintainers to assess.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• Obtain an explicit maintainer decision on the built-in scanner/runtime surface.
• Rebase onto current main and post a redacted runtime-image scan transcript for the final head.

Risk before merge

• [P1] Merging introduces a new built-in scanner that can execute scanned skill code inside the runtime sandbox, plus an externally sourced Python tool and test dependency; green unit checks do not settle whether that is an acceptable default runtime responsibility.
• [P1] The PR is currently dirty against main, so the final three-way merge and runtime image behavior need a rebase and refreshed validation before any merge decision.

Maintainer options:

1. Review the execution boundary before merge (recommended)
   Require maintainer approval of the new Docker-installed tool and its ability to run scanned skill code, then rebase and repeat the redacted runtime scan proof.
2. Keep Relyable outside the built-in runtime
   Do not merge the new scanner dependency if the project should remain limited to established security scanners and avoid this additional code-execution surface.

Next step before merge

• [P2] A maintainer must choose whether to support this new built-in execution-capable scanner and runtime dependency; there is no safe mechanical repair to queue before that choice.

Maintainer decision needed

• Question: Should ClawScan ship Relyable as a built-in, Docker-installed scanner that can execute scanned skill code under the sandbox, or should this remain an external optional integration?
• Rationale: The repository policy permits command-backed scanners and raw evidence, but adding a new execution-capable third-party runtime dependency changes the product boundary and cannot be resolved mechanically from tests alone.
• Likely owner: maxgris — The available commit history ties this contributor directly to the adapter and its security-boundary design; an established current-main area owner was not available from the review evidence.
• Options:
  • Sponsor the built-in integration (recommended): Accept Relyable as a supported scanner, request a rebase plus refreshed runtime proof, and merge only after the final image and sandbox boundary are reviewed.
  • Keep it external: Decline the built-in/runtime dependency while inviting documentation or profile guidance for operators who choose to install Relyable separately.

Security
Needs attention: The patch has a deliberate but material security and supply-chain boundary: it installs a new external tool into the runtime and enables it to execute scanned skill code under Docker, requiring maintainer acceptance rather than a line-level defect finding.

Review details

Best possible solution:

Decide whether functional re-derivation belongs in ClawScan’s built-in scanner catalog; if accepted, rebase the branch, retain explicit per-scanner credential forwarding and Docker-only execution acknowledgement, then validate the rebuilt runtime with a redacted live scan.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a bug report: this is a new scanner integration, and the PR body supplies a concrete claimed Docker-mode run rather than a failing current-main reproduction.

Is this the best way to solve the issue?

Unclear until maintainers choose the product boundary. The adapter follows the repository’s command-backed, raw-evidence pattern, but adding an execution-capable scanner to the built-in runtime is not the only viable delivery model.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a37f49f1b09f.

Label changes

Label changes:

• add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
• add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body includes an after-fix Docker-sandbox evidence excerpt from the adapter, states the exact image installation path, and describes both a diverged-skill result and a no-oracle demo result; refreshed redacted terminal proof is still appropriate after rebasing.
• remove rating: 🦐 gold shrimp: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
• remove status: 👀 ready for maintainer look: Current PR status label is status: ⏳ waiting on author.

Label justifications:

• P2: This is a bounded but consequential scanner/runtime feature requiring explicit maintainer product direction rather than urgent incident handling.
• merge-risk: 🚨 security-boundary: The built-in adapter can invoke a tool that executes scanned skill code and adds a third-party tool installation to the sandbox runtime.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body includes an after-fix Docker-sandbox evidence excerpt from the adapter, states the exact image installation path, and describes both a diverged-skill result and a no-oracle demo result; refreshed redacted terminal proof is still appropriate after rebasing.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes an after-fix Docker-sandbox evidence excerpt from the adapter, states the exact image installation path, and describes both a diverged-skill result and a no-oracle demo result; refreshed redacted terminal proof is still appropriate after rebasing.

Evidence reviewed

Security concerns:

• [medium] Review the new execution-capable runtime dependency — docker/clawscan-runtime/Dockerfile:27
  The Docker image installs Relyable from an external Git revision, and the adapter grants --allow-host-exec whenever Docker sandbox mode is selected. The credential-forwarding hardening is helpful, but maintainers must still accept this new untrusted-skill execution and supply-chain surface.
  Confidence: 0.93

What I checked:

• Proposed public scanner surface: The branch registers relyable as a command-backed built-in scanner, documents it in both scanner catalogs, and adds it to the accepted scanner ID roster. (internal/runner/scanner_registry.go:273, 0479449e88a6)
• Execution and secret-boundary design: The adapter passes --allow-host-exec only in Docker sandbox mode; the follow-up commit narrows sandbox forwarding to explicit RELYABLE_ variables rather than generic provider keys. (internal/runner/relyable_scanner.go:28, 0479449e88a6)
• Runtime supply-chain change: The runtime image installs Relyable from an external Git revision and adds pytest to its isolated tool environment, so acceptance expands the shipped runtime and execution surface. (docker/clawscan-runtime/Dockerfile:4, 0479449e88a6)
• Feature provenance: The available PR history shows the initial adapter in 280e958f and the security follow-up in 0479449e, which specifically removed automatic forwarding of generic LLM provider keys. (internal/runner/scanner_registry.go:273, 0479449e88a6)
• Current review state: The PR is not merged, its GitHub mergeability state is dirty against an older base, and the previous completed ClawSweeper cycles found no line-level defect but required maintainer review for product direction. (a37f49f1b09f)

Likely related people:

• maxgris: Authored both available commits, including the explicit sandbox-environment hardening follow-up; no current-main feature-history owner could be established from the available read-only inspection. (role: proposed integration author; confidence: medium; commits: 280e958f0128, 0479449e88a6; files: internal/runner/relyable_scanner.go, internal/runner/scanner_registry.go, docker/clawscan-runtime/Dockerfile)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (3 earlier review cycles)

• reviewed 2026-07-08T22:13:42.196Z sha 280e958 :: needs maintainer review before merge. :: none
• reviewed 2026-07-08T22:25:26.402Z sha 0479449 :: needs maintainer review before merge. :: none
• reviewed 2026-07-08T22:31:01.939Z sha 0479449 :: needs maintainer review before merge. :: none

[maxgris]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

[maxgris]

Rebased onto current main (a37f49f); the branch is mergeable again. New head is da490d0.

Resolution notes:

• Kept the updated catalog wording and supportsPlugins: true on skillspector from the plugin-support work. The relyable adapter does not set supportsPlugins, so plugin targets get the registry's explicit per-kind skip rather than an untested code path.
• Dockerfile: kept the new SKILLSPECTOR_REF and AIG_SKILL_SCAN_VERSION pins and re-added RELYABLE_REF. The pinned ref f7e1712 is still the current head of veriker/relyable, so the pin is unchanged.

Refreshed validation on the rebased head:

• gofmt clean, go build and full go test ./... green.
• Rebuilt docker/clawscan-runtime from the merged Dockerfile (not a derived image); relyable-scan, aig-skill-scan, and skillspector all resolve in the image.
• Re-ran the sandbox-mode scan through the rebased binary against the drifted-adder fixture (--scanner relyable --sandbox docker, rebuilt image): grade self_spec (tier S-B), verdict DIVERGED, tool CONTRADICTS, llmLaneAvailable: false (no key was set for this run, so the LLM lane was never exercised).

Command and run summary (paths redacted):

$ clawscan fixtures/drifted-adder --scanner relyable --sandbox docker \
    --sandbox-image clawscan-runtime-rebased:da490d0 --output artifact.json
targets: 1
scanner_completed: 1
scanner_failed: 0
scanner_skipped: 0
issues_found: 0
errors: 0
full_results: artifact.json

The relyable scanner block, from scanners.relyable.raw inside artifact.json (the file's own top level is the aggregate clawscan-run-v1 envelope):

{
  "schemaVersion": "relyable-scan-v1",
  "axis": "functional-rederivation",
  "scanner": {"name": "relyable", "version": "0.2.0"},
  "allowHostExec": true,
  "llmLaneAvailable": false,
  "skills": [{
    "skill": "drifted-adder",
    "grade": "self_spec",
    "selfSpecTier": "S-B",
    "verdict": "DIVERGED",
    "tools": {"drifted-adder:add": "CONTRADICTS"}
  }]
}

allowHostExec: true is the adapter's own flag for the Docker sandbox being the disposable host, per the existing design; nothing executed outside the container.

[maxgris]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

• State: Review in progress
• Detail: The exact-review queue leased this run; Codex is reviewing the item.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/29873640413
• Updated: 2026-07-21T22:24:37.721Z

[maxgris]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

• State: Review in progress
• Detail: The exact-review queue leased this run; Codex is reviewing the item.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/29882758745
• Updated: 2026-07-22T01:19:20.840Z

[clawsweeper]

ClawSweeper status: review started.

I am starting a fresh review of this pull request: feat: add relyable scanner adapter (functional re-derivation evidence) This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.