doc: add AI guidance for security report triage

[RafaelGSS]

As discussed in the Node.js Collaborator Summit. This is another attempt to reduce the AI-Sloop and reduce the amount of duplicate invalid reports we receive regularly (a more polite way to express what @panva said in the meeting 😄)

cc: @nodejs/security @nodejs/tsc

[nodejs-github-bot]

Review requested:

• @nodejs/tsc

[mcollina]

This will override any user setting.

I have an AGENTS.md locally. If we add this, it should include some more details about the project so that it can build/run it successfully.

[joyeecheung]

Note that there is not a uniformed way for people to build Node.js locally e.g. ninja vs make.

This file is currently too long. AI agents are known to disobey these instructions in the root files especially when they are long and the agents lose attention on them as they fill the context with other information. For telling them what NOT to do, these files are the wrong place.

Potentially a better solution might be to add an agent skill instead so that we can tell the humans to tell their agents to use the skill when finializing their security work (preferably at the end of their workflow to make sure agents obey). Agents will usually follow the skill better when it's a specific workflow. It also prevents interference with normal development flows.

[tniessen]

Thanks Rafael, it'd be great to reduce the inflow of AI slop in all areas. Is it necessary to add several new files to the root directory of the repository? It's already quite bloated. (I'm honestly asking, I am not familiar with these conventions.)

[RafaelGSS]

@mcollina @joyeecheung @tniessen

I was investigating the usual approach for someone who simply asks an LLM tool to "find vulnerabilities in this source code". Due to the number of files in this repository, some are ignored or have low impact on vulnerability analysis (such as the guidelines in doc/*). When manually consulting different LLM models on different framework platforms (Claude, Windsurf, Cursor), all indicated that the AGENTS.md file would be the best strategy to ensure that the LLM models read it.

Note that the goal is simply to find a way to ensure that people performing source code fuzzing using LLM models take into account our threat model and all the already documented restrictions. If anyone has a better solution, I would be happy to hear it.

[joyeecheung]

I think a lot of information captured here is not really about "how you find vulnerabilities" but "how you verify what you find is actually a vulnerability", for that purpose I don't think AGENTS.md etc. which appear at the beginning of the context would be respected much. For better alignment I think we can just offer a verify-vulnerability skill that we tell the reporters to use before they actually submit the bugs to us & also we can reuse it for initial triage of the vulnerability? If they just send their agents without taking the verification into account, it will just waste their tokens when they need to rule out the false positives at verification at their own expense. We can demand that if the report is assisted by AI, they must use the skill to produce a report with a specific format to support their claims. And we can refine the skill as much as we could to rule out the false positives.

[RafaelGSS]

I think we can just offer a verify-vulnerability skill that we tell the reporters to use before they actually submit the bugs to us & also we can reuse it for initial triage of the vulnerability?

I think it won't hurt to have this skill; it's actually a good idea. But I doubt it would be used. How different is it from the "read the SECURITY.md" prompt? I ask most of the reporters who are in contact with me about their workflow for finding vulnerabilities (for the ones that aren't applicable), and some of them just say: "find vulnerabilities and fill the report on this format $MARKDOWN_FROM_H1"

[joyeecheung]

"find vulnerabilities and fill the report on this format $MARKDOWN_FROM_H1"

I think we can just force them to do

"on this format $NODE_SECURITY_REPORT_FORMAT"

where the format enforces verification, and we provide a skill for the verification in the template. Their agents will just pick it up. Even if their agents don't pick it up, we can just reuse it with other agents, and potentially automate it if H1 supports it (let the agent optimistically close when it detect any sign of false positives or the lack of verification, which will motivate the reporter to double check). Also if they use it, it doesn't hurt to have another agent to verify it independently again.

[RafaelGSS]

I think we can just force them to do

How? I thought the AGENTS.md would be the only way to force it.

[joyeecheung]

How? I thought the AGENTS.md would be the only way to force it.

Through HackerOne report templates? AGENTS.md are pretty weak for enforcing general guidelines especially if it's meant to be enforced at the end of the workflow. If the requirement is e.g. a concrete checklist in the template, it has a much higher chance of being enforced because it then becomes a concrete task, no matter for humans or for agents. If the report does not conform to the template we close it automatically.

On the other hand, the security information in AGENTS.md is going to be meaningless bloat for non-security contributions which I suspect is what >95% of the agents that people use actually do for this code base, which means we may be trading having to deal with more spammy and low quality non-security PRs/issues for slightly less spammy security issues. I think we should avoid having to do this choice.

[RafaelGSS]

Through HackerOne report templates?

We already mentioned most (if not all) of the important files on HackerOne report documents (https://hackerone.com/nodejs), but this doesn't work. The report template https://hackerone.com/nodejs/submission_form is generally good. I'm less confident that adding checkboxes for: "Have you read the SECURITY.md?", or "Have you checked if the report isn't already listed as a known issue?" would change anything.

I have enough objections to this PR. I will figure out another alternative.

[joyeecheung]

I'm less confident that adding checkboxes for: "Have you read the SECURITY.md?", or "Have you checked if the report isn't already listed as a known issue?" would change anything.

I agree that kind of generic question wouldn't work well, but I think more concrete checkboxes would be a lot more effective.

Our current report template look like this

> NOTE! Thanks for submitting a report! Please replace *all* the [square] sections below with the pertinent details. Remember, the more detail you provide, the easier it is for us to triage and respond quickly, so be sure to take your time filling out the report!

**Summary:** [add summary of the vulnerability]

**Description:** [add more details about this vulnerability]

## Steps To Reproduce:

(Add details for how we can reproduce the issue)

  1. [add step]
  2. [add step]
  3. [add step]

[Also, if you were able to find the piece of code responsible for this issue, please add a link to it in the source repository.]

## Impact: [add why this issue matters]

## Supporting Material/References:

  * List any additional material (e.g. screenshots, logs, references, commits, code examples, etc.).

I think we could consider adding

## Verification

> Use this agent skill <link> to verify that the bug is a legitimate vulnerability instead of a regular bug. If it's a regular bug, submit to the public issue tracker instead:

- [ ] It does not require execution of untrusted/unaudited/intentionally malicious code.
- [ ] It does not require compromising the operating system/file system where the Node.js application is run
- [ ] (other most common cause of false positives)

[RafaelGSS]

I guess we can update the current document to:

@@ -1,5 +1,11 @@
 > NOTE! Thanks for submitting a report! Please replace *all* the [square] sections below with the pertinent details. Remember, the more detail you provide, the easier it is for us to triage and respond quickly, so be sure to take your time filling out the report!
 
+> **IMPORTANT:** Before filing this security report, please ensure you have:
+> 1. Read [`SECURITY.md`](https://github.com/nodejs/node/blob/main/SECURITY.md), especially the threat model and vulnerability criteria
+> 2. Reviewed the relevant API documentation in [`doc/`](https://github.com/nodejs/node/tree/main/doc)
+> 3. Verified this behavior is not documented as expected, trusted, or outside the Node.js threat model
+> 4. If using automated tools, cited the exact documentation sections that support your assessment
+
 **Summary:** [add summary of the vulnerability]
 
 **Description:** [add more details about this vulnerability]
@@ -16,6 +22,16 @@
 
 ## Impact: [add why this issue matters]
 
+## Documentation Review:
+
+Please confirm you have reviewed the following and explain how this behavior conflicts with documented guarantees:
+
+  * [ ] Relevant API documentation in [`doc/`](https://github.com/nodejs/node/tree/main/doc)
+  * [ ] [`SECURITY.md`](https://github.com/nodejs/node/blob/main/SECURITY.md) threat model
+  * [ ] Common false positives: [`permissions.md`](https://github.com/nodejs/node/blob/main/doc/api/permissions.md), [`wasi.md`](https://github.com/nodejs/node/blob/main/doc/api/wasi.md), [`cli.md`](https://github.com/nodejs/node/blob/main/doc/api/cli.md#warning-binding-inspector-to-a-public-ipport-combination-is-insecure)
+
+[Cite the exact document sections that support your assessment and explain why this behavior is a vulnerability despite the documentation]
+
 ## Supporting Material/References:
 
   * List any additional material (e.g. screenshots, logs, references, commits, code examples, etc.).