Skip to content

fix(capability): detect unlabeled provider tokens in the outbound scan - #7636

Merged
huangruiteng merged 3 commits into
loopx-project:mainfrom
JasonBuildAI:codex/agent-t7q5
Oct 29, 2026
Merged

huangruiteng merged 3 commits into
loopx-project:mainfrom
JasonBuildAI:codex/agent-t7q5

Conversation

@JasonBuildAI

@JasonBuildAI JasonBuildAI commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Goal And Delivered Outcome

Author Declaration

  • Written by: model_agent (DeepSeek, DeepSeek Harness), at the account owner's direction and under their review; the owner set the contribution constraints and reviewed the change.

Implemented against

Criterion (spec clause) Disposition Symbol / path Test or command
Every provider token shape the canonical owner recognizes is reported by the outbound scan implemented PROVIDER_TOKEN_SHAPE_PATTERN_SOURCE, RULES["provider_token"] test_a_provider_token_with_no_label_is_not_certified_clean (9 shapes incl. JWT)
The documented --scan-root route rejects a credential-bearing file implemented scanner.scan_paths test_provider_token_in_an_env_file_on_disk_fails_the_whole_scan
The receipt keeps its content-free shape (relative location, line, ***) implemented ScanResult.to_public_dict same test asserts the root path is absent and sample_masked == "***"
Clean prose, placeholders and env references still pass implemented word boundaries and length floors test_provider_shape_does_not_flag_the_prose_and_labels_it_must_pass
The documented over-match (hyphenated slug starting with an issuer prefix) is stated, not left implicit implemented rule table same test, second half
A labelled provider token still fails closed; hit_count is not a per-credential count implemented _scan_one collects one hit per matching rule test_a_labelled_provider_token_still_fails_closed
The shared rule's text is unchanged for its existing callers implemented SECRET_LIKE_SURFACE_PATTERN differential run, see Validation
  • Self-check before submission: ran the focused capability suite, ruff, git diff --check, the three public-safe text owner/parity suites, the UTF-8 and CLI-manual guard tests, the semantic-vocabulary advisory, loopx canary premerge --from-git-diff, the repo's own loopx check on the changed test file, and a whole-worktree before/after scan. Ran a differential comparison of the extracted pattern against the pre-change pattern over every tracked source line. Disclose one pre-existing Windows-only test failure in the touched file (chmod-based unreadable-file case) reproduced at base, and one pre-existing local-registry failure in an unrelated smoke.

Scope And Continuation

  • Completed scope and remaining work: the prefixed provider-token class in the outbound scanner only. This does not add label-free short values, does not change scan selection, decoding, receipt shape or exit-code contract, and does not touch loopx check's LEAK_RULES.
  • Slice boundary / successor: complete within this scope; no successor needed. Two adjacent gaps are deliberately out of scope and stated so a reviewer does not read this as a wider repair:
    1. loopx check --scan-path (the LEAK_RULES gate in loopx/contract.py) still misses seven of these nine shapes. Widening it is a policy decision, not a mechanical one: the shared SECRET_LIKE_SURFACE_PATTERN matches 566 lines across the 6,558 UTF-8-decodable tracked files (measured with strict UTF-8 decoding, search() per line), and those are overwhelmingly legitimate source, so reusing it there needs its own false-positive review. This PR therefore does not close the loopx check half named in issue [Bug]: the builtin public-safe-outbound scan certifies GitHub, AWS, Slack, Google, npm, PyPI and Stripe credentials as public-safe #7635; it stays open for that successor.
    2. The inherited AWS/Slack prefix sets are narrower than the issuer families (ABIA/ACCA/AROA/… and xapp- are not covered). Those prefixes live in the canonical owner and widening them changes every consumer of SECRET_LIKE_SURFACE_PATTERN, which is a separate review.

Validation

  • Tested revision: a428f2a65 (re-verified after the review follow-ups below)
  • Run state: finished
  • Input classes: synthetic
Check kind Result Public-safe evidence / limitation
unit passed pytest loopx/capabilities/public_safe_outbound/tests -q: 50 passed, 4 skipped, 1 failed. The failure is test_unreadable_requested_file_fails_closed (a chmod(0o000) case that cannot deny read on Windows) and reproduces identically at base.
regression_parity passed All four new tests were re-run against the base scanner file verbatim: 4 failed at base, 4 passed at head.
regression_parity passed Pattern extraction differential: SECRET_LIKE_SURFACE_PATTERN.pattern is byte-identical before/after, and searching every tracked source line plus adversarial samples produced zero verdict differences. An independent reviewer separately compared the compiled objects (686 chars, flags 34, groups 0) and 2,048,874 lines with zero search()/group(0) differences.
integration passed pytest tests/control_plane/test_public_safe_text_classifier.py tests/control_plane/test_public_safe_text_owner_parity.py -q: 265 passed.
real_entrypoint passed python -m loopx.capabilities.public_safe_outbound.scan_cli --scan-root <dir> --format text over one file per shape: base PASS: public-safe exit 0; head [provider_token] … for all nine, exit 1.
manual passed Whole-worktree --scan-root on the head tree: 172 hits at base → 177 at head. New locations are provider_token at examples/control_plane/goal-artifact-lifecycle-projection-smoke.py, tests/test_chat_activity.py, tests/fixtures/control_plane/reliability_diagnostics_public_safety_v0.json, plus this PR's own test_scanner.py; the first three are deliberate synthetic fixtures that already had other hits at base. No location lost a hit. loopx check on the changed test file reports ok: true, blocking_hits: [], warnings: [].
static passed ruff check loopx/public_safe_text.py loopx/capabilities/public_safe_outbound/; git diff --check; loopx canary premerge --from-git-diff --git-diff-base upstream/main (direct checks and all selected canaries passed, public boundary clean).
  • Coverage and gaps: The rule, the end-to-end file route, the passing edge, the stated over-match and the two-rule labelled case are each pinned. Not qualified here: hosted CI (not run) and any non-UTF-8 or BOM-less candidate, whose decoding behavior this change deliberately leaves alone.

Frontend / Visual Evidence

  • UI impact: none

Type of Change

  • Bug fix

LoopX Area

  • Capability or extension (providers, adapters, skills)

Boundary Checklist

  • Neither the diff nor this PR body/comments/attachments disclose private state, credentials, raw traces or verifier output, internal links, or local machine paths.
  • I did not duplicate maintainer-owned benchmark work.
  • I kept the change scoped to the linked issue/task.
  • No paired RFC changed.
  • I completed the visual evidence section (UI impact none).
  • Every commit includes a DCO Signed-off-by trailer.

AI Disclosure

AI disclosure: this change was drafted with AI assistance at my request, under my direction and review.

The builtin public-safe-outbound credential table recognized an Ark key, an
`sk-`/`Bearer` value, an `AKLT` value, a private-key block and a quoted
`api_key=` assignment. Every other provider credential therefore returned
`ok: true` with exit 0 -- a positive public-safe certification -- for the eight
most common issuers: GitHub (`ghp_`, `github_pat_`), AWS (`AKIA`, `ASIA`),
Slack (`xoxb-`), Google (`AIza`), Stripe (`sk_live_`), npm and PyPI. Three of
those are the values `.env`, `~/.npmrc` and `~/.aws/credentials` actually hold,
and `.env` is exactly the credential-named file the scan's deny-list selection
promises to review.

The canonical private-text owner already carried all nine shapes in
`SECRET_LIKE_SURFACE_PATTERN`, so the two owners disagreed and the gate that
certifies published content was the narrower one (the drift class in loopx-project#5136).

Extract that alternation into `PROVIDER_TOKEN_SHAPE_PATTERN_SOURCE` and consume
it from a new `provider_token` rule. The extracted text is byte-identical to
the inline block it replaces: a differential run over 6,024,515 repository
lines and fourteen adversarial samples found zero verdict differences, so
existing callers of `SECRET_LIKE_SURFACE_PATTERN` keep their behavior.

No scan-selection, decoding, receipt-shape or exit-code contract changes.

Fixes loopx-project#7635

Signed-off-by: JasonBuildAI <jasonbuildai@gmail.com>
Three points from the independent review, all within this change's scope:

- The prose test claimed the length floors keep documentation out, but its
  fixtures avoided the actual over-match edge. `pypi-django-rest-framework` and
  `xoxb-slack-app-configuration` are reported. Pin that direction explicitly:
  it is the capability's documented "prefer a false hit over a miss" boundary,
  not something a reader should discover by accident.
- The duplicate-hit test asserted a guarantee that does not hold. `_scan_one`
  collects one hit per matching rule, so `api_key='<provider token>'` yields
  both `provider_token` and `api_key_assignment`. Replace it with a test that
  states the real contract: the verdict and the content-free receipt, with
  `hit_count` explicitly not a per-credential count.
- The README advertises the JWT shape and the rule matches it, but no test
  pinned it. Add it to the shape table.

Also replace the stale "first matching rule wins per file" comment on the rule
table, and assemble the `bearer` word in the prose fixture so this test file no
longer adds two advisory warnings to the repository's own `loopx check` scan.

Signed-off-by: JasonBuildAI <jasonbuildai@gmail.com>
The two new docstrings quoted the very strings the rule reports, so this test
file went from one to three hits under the capability's own scan. They describe
behavior; they should not read as candidates. Rewording keeps the pin and
leaves the directory's scan at its three pre-existing private-path hits.

Signed-off-by: JasonBuildAI <jasonbuildai@gmail.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; OpenAI gpt-6.1-sol; reasoning effort max; runtime_reported.

Reviewed exact head: a428f2a65055a82516379e7db5eaa32fba830700.

动机 / Motivation

贡献者在公开提交前运行现有 outbound 扫描时,无标签的常见服务商令牌原来会获得“无命中、退出 0”的结果,可能据此把敏感内容公开。本次修复让同一批候选内容产生脱敏命中并退出 1,同时保留普通文字通过、定位后修正并重新扫描的路径。改善范围是手动调用的 outbound 扫描,不涉及自动 CI 接入、所有凭证格式或其他扫描器。

改动思路 / Approach

沿用现有文本规则所有者:从共享正则中提取服务商前缀分支,再由 outbound 的规则表直接消费。共享正则编译后的内容、flags 和 groups 保持一致;扫描器新增一个诊断规则,命中后仍由原有结果对象和命令行返回结果。这样避免维护第二份前缀列表,也没有增加配置、持久状态或执行权限。当前 PR 的边界是已有扫描入口的检测遗漏修复和共享规则的等价提取。现有 Python 文本处理所有者足够,无需额外控制层或语言迁移。

基准规范为 loopx/capabilities/public_safe_outbound/README.md,revision 652f503842609e948e8ff832ae3977f23f79f28d,结合 #7635 的当前 Boundary。以下 criterion IDs 是本评审对已有条款的标识:

Criterion 条款与实证
PSO-CRED 选中内容含已支持的凭证形状时拒绝:10 个服务商代表形状旧版退出 0,新版退出 1。
PSO-RECEIPT 返回相对位置、原始行号和完全遮蔽样本;真实 CLI 的 JSON/text 结果不包含合成令牌或扫描根目录。
PSO-CONFIG 未启用本地配置时使用通用规则;显式不可读配置仍 fail closed;本地标记开关结果保持一致。
PSO-UNSCANNED 超限内容仍报告未扫描并拒绝,原有选择和解码边界保持不变。
PSO-CONSERVATIVE 沿用规范已有的保守识别取舍;普通文字、占位符、环境变量引用和短前缀通过,已知长前缀 slug 误报得到明确披露。
PSO-SHARED 现有共享文本消费者保持结果:编译正则完全相同,21 个相同输入的公开校验/压缩结果逐项一致。

具体改动 / Changes

四个文件合计 +134/-7。public_safe_text.py 提取既有 alternation,scanner.py 加入 provider_token 并修正“一条匹配规则一个命中”的注释;README 补充识别范围;四项测试覆盖无标签令牌、磁盘 .env、通过/保守误报边界和已带标签的双规则命中。hit_count 是规则命中数,不能解释为不同凭证数。

关键路径已按定义和调用者核对:共享规则 → 扫描循环 → 既有 CLI。同一合成夹具在不可变 base/head 上各执行 28 次真实 CLI 子进程,覆盖目录扫描、stdin diff、JSON/text、UTF-16 BOM、标签重叠、超限和配置失败。修正内容后连续两次重扫均退出 0;扫描没有修改输入或另建恢复状态。

独立验证:本次 macOS/Python 3.14.8 的 capability + classifier + owner-parity 三组测试共 320 passed、0 skipped;同三组在不可变 base 为 316 passed、0 skipped。把新增测试原样复制给 base 时四项均失败,head 中四项通过,证明它们能抓住原缺陷。Ruff、Python 3.11.17 对变更 Python 文件的编译、diff 检查均通过。以已获取主干 3344f540f6f4992d95de1c04d50ba586d64deddd 为目标的原生 premerge 完成 6 项直接检查和 15 项选中检查,无失败;CQR cqr_ddae4ae473ce480836e6 已读回 valid。这里报告独立本地执行结果,远端 CI 未查询或等待。

对主干的风险 / Risks

最强反例是正常文本恰好拥有足够长的服务商前缀,导致扫描拒绝并增加人工处理。这个成本确实存在:PyPI/Slack 风格 slug 可能误报;它符合修改前规范的保守策略,且新增测试和 README 已明示。恢复仍是检查相对位置、移除或改写候选敏感内容后重新扫描,不会静默绕过失败。五类普通负例和重复恢复路径已通过,但有限夹具不能证明不存在其他误报或漏报。

共享消费者的兼容性通过完全相同的编译规则及 21 个真实函数结果核对;当前目标主干上的两个生产文件也未偏离所用 base。PR 没有 default-off 声明:内置能力可用与操作者实际运行是两回事,新规则在显式调用扫描时生效。没有改变调度、授权、持久状态、机器指令或自动安装路径。loopx check 的 LEAK_RULES、更广的 AWS/Slack 家族和 BOM-less 旧编码继续维持原边界,不能把本评审读成对它们的安全保证。

作者披露的 Windows chmod 测试失败仍属于历史记录;本机 320 项通过不证明 Windows 原生环境已修复或已验证。完整发布包、安装态和未运行平台也没有在此获得资格。若新规则产生不可接受的误报,可回滚本次四文件变更;无需迁移数据。

我的整体评价 / Verdict

该差异完成 #7635 当前限定的 outbound 规则表修复,真实用户入口能够从漏检变为拒绝,并可在修正后继续工作。没有发现需要阻止该提交的正确性或权限问题。共享定义复用和保留原命令行路径是本次足够且可回滚的边界;额外包装或重复规则没有收益。长程维度的结论是既有重复使用/恢复行为得到保留,没有声称节省总成本或证明长期效益。

视觉检查不适用:实际改变的是 CLI 检测规则和 README 规则列表,没有渲染页面或首屏布局变化。未验证范围:没有新增 UI 旅程;安装态、远端 CI 和 Windows 原生运行未验证。审批结论与合并权限分开,本评审不执行合并。

English verdict: APPROVE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: the builtin public-safe-outbound scan certifies GitHub, AWS, Slack, Google, npm, PyPI and Stripe credentials as public-safe

3 participants