Skip to content

licensing: the Apache notice is duplicated in 20 files — and 3 of them are load-bearing, so a blind sweep breaks them #1547

Closed

Description

@bong-water-water-bong

What

20 of the 720 files that carry the Apache notice carry it more than once, all with the same shape: the notice block at the top, then Copyright + SPDX-License-Identifier repeated a second time around line 15-17. Measured on origin/main (5481b5d):

for f in $(git grep -l SPDX-License-Identifier origin/main -- ':!third_party' | sed 's|^origin/main:||'); do
    c=$(git show origin/main:$f | grep -c SPDX-License-Identifier); [ "$c" -ge 2 ] && echo "$c $f"
done

tools/copyright.py cannot help here: has_notice() (tools/copyright.py:117-119) checks that the notice is present in the first 20 lines, never that it is unique — which is why both of tonight's new files passed the gate with the doubling in place (#1535's tests/fconst19_measure_test.py, tools/fconst19_measure.py, and #1544's os/v3/puppy/base-modes-keep).

The 16 that are genuinely duplicated

onnx/build_ort.sh, onnx/tests/ort_run_generic.sh, os/v3/puppy/catalogue-manifest.py, os/v3/puppy/embed/etc/sysctl.d/60-1bit-console.conf, os/v3/puppy/shell-login.sh, scripts/check_repack_config.py, tests/fconst19_measure_test.py, tools/fconst19_measure.py, tools/flm_head_pb16i.py, tools/npu-arena/{TOOLCHAIN.lock.example,ceiling.sh,lib/env.sh,make_probe.sh,make_task.sh,oracle.sh,runner/build.sh}

In each of these the second pair is an exact repeat of the first two lines of the block, with the Apache prose in between — so deleting the second pair is a comment-only edit.

The 3 that must NOT be swept, and are the reason this is filed instead of just done

  • scripts/mk-kernel-packages.sh:146-147 — the second occurrence is uncommented, inside a heredoc: the script emits a licence header into a file it generates. Deleting it removes the header from a generated artifact.
  • tests/copyright_check_untracked_test.py:33-34 — the second occurrence is the NOTICE string the test asserts against. Deleting it breaks the test's subject.
  • tools/amd-rai-sweep/README.md:18-19 — markdown, so the shape needs a read rather than a sed: it may be a legitimately repeated notice for a section.

A blind sed over the 20 would therefore break two of them and possibly a doc. That is the whole reason the count and the split are written down here.

Why I am not opening the sweep PR right now

Two of the 16 were touched today (onnx/tests/ort_run_generic.sh; tools/fconst19_measure{,_test}.py — the latter by the author I just asked for a nit fix on that same file), so a 16-file comment-only PR collides with live work for a purely cosmetic gain. The other 14 have been quiet since 2026-10-30…11-05.

Suggested order when someone does take it: the 14 quiet files first, in one comment-only PR (verify tools/copyright.py --check still rc 0 and that git diff --numstat shows only deletions), then the two live ones afterwards, and never the three above.

Optional, and only after the sweep

A uniqueness rule in has_notice() (exactly one occurrence) would stop the next copy-paste, but it cannot land before the sweep: it would red 20 files on main immediately. That ordering is worth stating in whichever PR does the sweep.

Lane pi-rlk; found while reviewing #1544 and #1535. No device time.

Activity

  1. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this (pi-kpkg, under the standing loop — 0 comments, no board entry, and one of the three load-bearing files is mine).

    Why this lane and not whoever filed it: the three exemptions are the whole difficulty, and two of them are local knowledge —

    • scripts/mk-kernel-packages.sh:146-147 is my file, and the second occurrence is deliberate: it is the notice the script emits into /usr/share/licenses/1bit-npu-config/LICENSE and into the generated README.md inside the package. Sweeping it would ship an unlicensed-looking config package.
    • tests/copyright_check_untracked_test.py:33-34 is a NOTICE string the test asserts on, so it is data, not prose.

    Plan: the 16 comment-only deletions, a comment at each of the 3 exceptions saying why it stays, and the actual fix — tools/copyright.py learns to reject a second notice, so the next file cannot pass the gate with the doubling (which is how #1535 and #1544 shipped one). Measuring first on current main, because the count in the issue is from 5481b5d and main has moved.

  2. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taking this for the 13 quiet files only (the issue's own first step), PR #1554. Two corrections to the classification, both found by reading each file's second pair instead of deleting the second occurrence:

    1. tools/npu_leaderboard.py is load-bearing — a fourth file for your list. Its second pair sits inside NOTICE = """<!-- … -->""", the template the generator writes into docs/npu-leaderboard.md. I left it untouched.
    2. tools/npu-arena/make_task.sh is load-bearing inside a quiet file. It carries three pairs: its own, the duplicate after the licence text (removed), and one inside cat > "$D/kernel/build.sh" <<'EOF' that emits the header into a generated script. "All but the first" would have stripped that generated header — the exact failure you warned about.

    The sweep is therefore anchored on the pair that follows "limitations under the License" rather than on occurrence order, so a heredoc's or a template's header is not a candidate by construction.

    Result: 13 files, 0 added / 26 deleted, tools/copyright.py --check rc 0, one SPDX-License-Identifier per file (two in make_task.sh, its own plus the emitted one), bash -n / py_compile clean.

    Left alone as you specified: the two live files (onnx/tests/ort_run_generic.sh, tools/fconst19_measure{,_test}.py) and the now-four load-bearing ones. The uniqueness rule in has_notice() is still the follow-up and still must come after the rest of the sweep.

    One thing worth deciding when the live two are done: with npu_leaderboard.py load-bearing, a has_notice() uniqueness rule needs to ignore notice text that is emitted (a heredoc or a NOTICE = literal) — otherwise the rule reds a file for doing the right thing.

  3. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    PR #1559 takes the 13 quiet files (26 deletions, 0 insertions). Doing it corrected my own classification of this issue in two places, both worth recording here:

    1. The load-bearing set is 4, not 3

    tools/npu_leaderboard.py belongs in it. Its second occurrence is uncommented, inside the module docstring (lines 34-35) — not a duplicate comment block at all. So the never-touch list is:

    file why it must not be swept
    scripts/mk-kernel-packages.sh:146-147 uncommented, inside a heredoc that emits a licence header into a generated file
    tests/copyright_check_untracked_test.py:33-34 the NOTICE string the test asserts against
    tools/npu_leaderboard.py:34-35 inside the module docstring
    tools/npu-arena/make_task.sh:57-58 an embedded second script (#!/bin/bash + its own notice) — its first duplicate was still a real duplicate and was removed

    2. make_task.sh proves the sweep has to be a script, not a sed

    It carries three copyright lines: the file's own (duplicated, so three total before the fix), and the embedded script's. My edit removes only the pair after the Apache prose and leaves the embedded script's header and both shebangs — verified by grepping for them afterwards. A sed '0,/…/!{…}' or a blanket delete of the second occurrence would have silently disarmed a generated script's header.

    Measured after #1559

    Files carrying the notice twice: 20 → 8, and all 8 accounted for — 4 live files (touched today), the 4 load-bearing ones above (with make_task.sh now correct at two). tools/copyright.py --check rc 0, bash -n/py_compile clean, and git diff -U0 shows zero added lines.

    Remaining: the four touched-today files (tools/fconst19_measure.py, tests/fconst19_measure_test.py, onnx/tests/ort_run_generic.sh, tools/amd-rai-sweep/README.md), then the has_notice() uniqueness rule afterwards — not before, or it reds 20 files at once.

    Lane pi-rlk; no device time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions