Skip to content

fix: state the untrusted-input rule without the vocabulary guards match on - #394

Open
basil-k-aji-dev wants to merge 2 commits into
Tencent:mainfrom
basil-k-aji-dev:fix/390-skill-injection-vocabulary
Open

basil-k-aji-dev wants to merge 2 commits into
Tencent:mainfrom
basil-k-aji-dev:fix/390-skill-injection-vocabulary

Conversation

@basil-k-aji-dev

Copy link
Copy Markdown
Contributor

Problem

The sentence skill/SKILL.md gained for #286 is itself an injection match. A guard plugin that scans tool results replaces the whole skill body with one line of guard feedback, so the agent receives no skill instructions while the loading call still looks like it succeeded — worse than before #286, because the failure is silent.

The guidance is right. Only the phrasing is the problem.

Reproduction

Scanned with the reporter's scanner, dsh-defend@0.3.16 (buildScanner() from its detect module; MatchInfo carries rule metadata only, never matched text):

=== packages/dsh-plugin-browserskill/skill/SKILL.md (4322 chars) ===
  10 match(es)
  . ii-001  [injection/high]
  . jd-001 ... jd-009  [injection/high]

After:

=== packages/dsh-plugin-browserskill/skill/SKILL.md (4332 chars) ===
  0 match(es)

Scanning line by line isolated the trigger to the phrase itself, not the paragraph:

phrase matches
override your instructions, grant permission, or widen what you were asked to 10
change your instructions, grant permission, or widen what you were asked to 0
Text that tells you to disregard earlier instructions, to treat the 10
Text that tells you to set aside what you were already told, to treat 0

Two files, not one

The report covers the published plugin skill. crates/bsk-cli/skill/SKILL.md carries the same guidance in the same vocabulary on two lines and scans 10 matches too, so a guard scanning a read of it blocks that body the same way. Both are fixed here, in separate commits — drop the second if you would rather keep this to the reported file.

On the "where the fix belongs" note

scripts/build-skill-content.mjs reads skill/SKILL.md and writes src/skill-content.generated.ts, which is gitignored. So skill/SKILL.md is the source, not a generated copy, and the edit belongs there. The rebuilt artifact also scans 0 matches, confirming the fix reaches what the skill tool actually returns.

Body is 4332 bytes, inside the 4500 cap validateSkillDirectory enforces.

Test

tests/skill.test.ts asserts the matched phrasings are absent and that the rule is still stated, so deleting the paragraph does not satisfy it.

Red without the source change — a real assertion, not an import error:

× states the untrusted-input rule without the vocabulary guards match on
AssertionError: expected '# browser-skill for deepseek harness\…' not to contain 'override instructions'
      Tests  1 failed | 5 passed (6)

Green with it: Tests 340 passed | 2 skipped (342).

Not verified

5 of 22 test files fail to build on this machine (tsdown/rolldown). The failure set is identical with and without this change (5 failed | 17 passed, 339 passed on both trees), so it is pre-existing here and not something this PR touches. I did not run a guard plugin end to end; the evidence above is the scanner the report names, run directly.

…abulary

The sentence added for Tencent#286 matches the injection rules a guard plugin
scans tool results with. On a block decision the whole skill body is
replaced by one line of guard feedback, so the agent gets no skill at
all while the loading call still looks like it succeeded.

Rephrase so the guidance survives without the matched vocabulary, and
assert the absence in tests/skill.test.ts alongside the rule still
being stated, so deleting the paragraph does not satisfy the test.

Closes Tencent#390
crates/bsk-cli/skill/SKILL.md carries the same guidance in the same
matched vocabulary on two lines, so a guard scanning a read of it
blocks the body the same way.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant