Skip to content

RFC 239: Policy on LLM assistance in contributions #239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jugglinmike wants to merge 4 commits into
base: main
Choose a base branch
from
+97 −0
Open

RFC 239: Policy on LLM assistance in contributions #239

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ba9ec3e
RFC 239: Policy on LLM assistance in contributions
jugglinmike May 20, 2026
3f4f8e0
Incorporate review feedback
jugglinmike Jun 2, 2026
8645bc2
Remove errant section
jugglinmike Jun 9, 2026
0bd38d7
Allow blockquotes for short segments
jugglinmike Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 97 additions & 0 deletions rfcs/llm_policy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# RFC 239: Policy on LLM assistance in contributions

## Summary

Introduce guidelines for acceptable use of large-language models when
contributing to web-platform-tests.

## Background

[#202 Set policy for LLM-generated
tests](https://github.com/web-platform-tests/rfcs/issues/202) includes evidence
for public interest in a formal policy for LLM usage in authoring contributions
to WPT.

The Chrome team is exploring applications of LLMs for detecting coverage gaps
and for filling those gaps with generated code. ([Project
repository](https://github.com/GoogleChromeLabs/wpt-gen), [April 2026
presentation](https://www.youtube.com/watch?v=9r0PBbJFLoM))

A few examples of policies on LLM use in FOSS contributions:

- permissive
- [ghostty/AI_POLICY.md at main · ghostty-org/ghostty](https://github.com/ghostty-org/ghostty/blob/main/AI_POLICY.md)
- [Policy about LLM generated code from PRs · Issue #28335 · opencv/opencv](https://github.com/opencv/opencv/issues/28335)
- [CONTRIBUTING.md: Guidelines relevant to AI-assisted contributions by gasche · Pull Request #14052 · ocaml/ocaml](https://github.com/ocaml/ocaml/pull/14052)
- [LLVM AI Tool Use Policy — LLVM 23.0.0git documentation](https://llvm.org/docs/AIToolPolicy.html)
- [Chromium AI Coding Policy](https://chromium.googlesource.com/chromium/src/+/4f44016dfbd4fcd890694c00d7f9ec6dcefe4955/agents/ai_policy.md)
- [Firefox AI Coding Policy](https://github.com/mozilla-firefox/firefox/blob/1f7030c8de8f2b349c7d91d7b5a3253c109a1cc1/docs/contributing/ai-coding.md)
- prohibitive
- [Code of Conduct ⚡ Zig Programming Language](https://ziglang.org/code-of-conduct/#strict-no-llm-no-ai-policy)
- [Getting Started - The Servo Book](https://book.servo.org/contributing/getting-started.html#ai-contributions)
Comment thread
gsnedders marked this conversation as resolved.

## Details

The following text describes the policy in full and will be maintained in a
dedicated document within WPT's `docs/writing-tests/` directory (which will be
referenced both from the project's `README.md` file and the
`docs/writing-tests/index.md` file):

> ### Guidelines for acceptable LLM use
>
> The use of large language models (LLMs) as tools to help author commits to
> this repository is allowed with the stipulations described below.
> Contributors who repeatedly fail to adhere to these guidelines may be banned
> from contributing to this project.
>
> #### Disclosure
>
> If LLMs are used as a significant input to a commit, authors are encouraged
> to include details about how they were used as part of the commit message in
> order to help review and future understanding of the code.
>
> Human-authored code discourse (e.g. issue descriptions, pull request
> descriptions, and responses to discussion threads) should not include
> LLM-generated content in the main text; any such content must be clearly
> labelled and placed inside a `<details>` element. A `<blockquote>` element
> may instead be used to designate text if it is no longer than one paragraph
> in length.
>
> #### Attribution
>
> All commits must be attributed to the human who is taking responsibility for
> them, regardless of LLM use.
>
> #### Understanding
>
> Every pull request must be initiated by one human. That person must author

@annevk annevk Jun 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be at least one? Don't we sometimes get team contributions?

@jgraham jgraham Jun 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All PRs are created by one user account which almost always means one person. Do you have an example of a PR that's attributed to a team rather than an individual?

@annevk annevk Jun 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That part makes sense, but I don't think the human uploading always knows about every line, if they are open sourcing a set of tests for instance that were developed internally over some period of time. E.g., I think some Opera contributions in the past were of this nature.

@jgraham jgraham Jun 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems like a case where we're trusting a third party repository as a source of commits (as we do for Gecko, Chromium, WebKit and Servo repositories today). It is true the policy there isn't quite made explicit anywhere, but I think we should deal with that separately compared to the policy for direct contributions to this repository.

@gsnedders gsnedders Jun 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have an example of a PR that's attributed to a team rather than an individual?

Maybe irrelevant detail, but you can create PRs whose head points at an organization's fork, though it is still a user account actually opening the PR.

@jugglinmike jugglinmike Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@annevk If you're satisfied by James' latest response, could you mark this as resolved?

@annevk annevk Jun 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I'm entirely satisfied, but I'm also not sure it matters much. I don't think we have to necessarily have a relationship with a third-party repository for someone to contribute tests from a third-party repository. Like if someone had an independent implementation of the URL standard and they and other contributors created a bunch of tests for it, I would not want to enforce that the person upstreaming those tests can defend and speak to them all themselves.

> the pull request description, understand every change proposed, and be
> prepared to engage in technical discussion regarding those changes.

## Risks

@gsnedders gsnedders May 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's worthwhile including at least a few more technical risks:

  1. Contributions of tests generated by an LLM closely looking at a specific implementation's code, matching that implementation, rather than the spec. (This is, of course, already an issue — but could inevitably become more of a problem if we get more, larger contributions.)
  2. Contributions not matching the spec at all. I've seen this mostly with trying to generate tests to assert ordering of things which end of using HTML's parallelism and HTML's event loops; that case is especially annoying because it can lead to flaky tests.

@jugglinmike jugglinmike Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me, those look like specific failure modes of LLMs. I think they'd be more helpful to elucidate what we mean by "low-value contributions" rather than as additional risks that are distinct from low-value contributions. I've just pushed a commit incorporating them in that way.

@gsnedders gsnedders Jun 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I think the (2) might be reasonable to class as "low value", I think (1) is really a different failure mode, and one that becomes a higher-risk with LLMs (by virtue of sheer volume).

@jugglinmike jugglinmike Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that (1) is distinct from (2), but I still don't see how it represents a meaningfully-distinct failure mode of this policy. Both (1) and (2) are second-order risks from defining a policy. They both flow from the premise that any such policy will encourage the use of tools that have these weaknesses. And they both have the same impact on our risk calculation here: we can't technically stop people, but an explicit policy formalizes a social contract.

That's what's keeping me from creating a new sub-section for over-fitting. It's easy to frame that as a separate risk, but it seems like we can only come to the same conclusion as we do for the "low-value contribution" risk: we accept it because we expect people to act in good faith and follow the policy.


### Discouraging volunteers

All but the most permissive policy is effectively another hurdle to
contributing to the project. Friction in the contribution process could deter
people who might otherwise volunteer their time to help improve the project.

In some sense, adding friction is the goal of this policy. New technology has
removed barriers which previously restricted unqualified individuals from
participation. Rather than introducing more restrictions on good-faith actors,
an ideal policy will buttress eroded structural barriers with more intentional
social ones.

### Encouraging low-value contributions

All but the most restrictive policy could be interpreted as an invitation to
take shortcuts which undermine the quality of contributions. Any permissive
policy might be taken as encouragement to rely on fallible tools (LLMs are
particularly susceptible to certain kinds of test-writing errors, such as
over-fitting and fabrication).

However, it will not be possible to strictly enforce any policy. It inevitably
falls on contributors to follow rules and for administrators to police
transgressions. Respect in public works projects is never guaranteed; policies
exist only to make expectations clear (this is the same dynamic that guides the
design and enforcement of codes of conduct).