Skip to content

Audit: remote content, written keys, look-alike names, whole-folder update diff - #23

Merged
devohmycode merged 1 commit into
mainfrom
feat/vetting-signals
Nov 10, 2026
Merged

devohmycode merged 1 commit into
mainfrom
feat/vetting-signals

Conversation

@devohmycode

@devohmycode devohmycode commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Four additions to the security audit, from what a review of a third-party skill should cover.

What changes

  • Content read from a URL (remoteContent, medium). A line where the skill fetches, reads or downloads something at a URL is reported: what is served there can change after the review, and pinning the skill to a commit does not pin it. A line that already matches "downloads and runs code" or "sends data" is reported under that sign only.
  • Secret keys written in a skill (hardcodedKey, medium): sk-…, ghp_…, github_pat_…, AKIA…, xox?-…, AIza…, private key blocks. The report shows the key cut to its first characters; placeholders are ignored.
  • Look-alike names (lookalike, medium). From the catalog: a name, or an owner's name for a skill of the same name, one character away from a skill installed at least 1,000 times and 20 times more. Shown by the install audit (CLI, tree, web), show, show_skill and the web preview, with the skill it resembles.
  • The update diff covers the whole folder. updateDiff compares the other text files too: files added, changed or removed are listed by update, shown in the tree's preview under the SKILL.md, and returned by outdated_skills as files_changed. New warning signs carry their file.

The three new labels are translated in the nine languages.

Limits

  • remoteContent is a pattern: a plain documentation link after "read" or "load" is flagged too. This changes one existing expectation: a docs table holding curl https://…/llms.txt used to be quiet and is now reported.
  • Look-alikes on a 1-in-40 sample of the catalog, measured with the first threshold (names of 5 characters or more): 12 of 973 skills, none in the 500 most installed, most of them honest neighbours (design-system / design-systems). The threshold was then raised to 8 characters; the rate was not measured again.
  • A file is reported as removed when the installed folder has a text file the repository no longer lists. Files the audit did not read are never counted as removed.

Tests

npm run typecheck and npm run build pass. npm test: 279 pass.

…diff every file on update

A skill that reads content at a URL is reported: that content can change after the review, and a pinned commit does not pin it. Keys written in a skill are reported, cut to their first characters. The catalog names the far more installed skill a name could be taken for. The update diff now compares the other text files of the folder, with their new warning signs.
@devohmycode
devohmycode marked this pull request as ready for review November 10, 2026 15:03
@devohmycode
devohmycode merged commit 311fc14 into main Nov 10, 2026
3 checks passed
@devohmycode
devohmycode deleted the feat/vetting-signals branch November 10, 2026 15:04
@github-project-automation github-project-automation Bot moved this from Backlog to Done in Skills-Atlas Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant