Skip to content

ci(fuzz): keep Extended's aggregate status meaningful while two fuzz targets are expected to fail #593

Description

@justin13888

The situation

PR #568 wires six parser-entry-point fuzz targets into extended.yml's fuzz matrix. Two of them
find real defects on their first run and are deliberately left failing: #563 (gamut-tiff
panics on SamplesPerPixel = 0) and #564 (gamut-dng sizes its raw buffer from declared
geometry, asking for 34 GB from a 780-byte file). Narrowing either target so its row went green
would be weakening a check to make a report green, so the rows stay red until the defects are
fixed. That decision is recorded on #568 and is not what this issue disputes.

The cost, which is real

extended.yml runs on every push to the default branch. Its aggregate conclusion is therefore
red on every push until #563 and #564 close. The containment is genuine — Extended is post-merge
and manual-dispatch only, the fuzz job is fail-fast: false, and no pull-request check is affected
— but the thing a human actually looks at is one tick per push, and that tick now says "red"
permanently for a reason unrelated to whatever was just pushed. A red signal that is always red
stops being read, and the next unexpected Extended failure arrives inside a status nobody
believes.

Marking or skipping the two rows is not the answer: a row that reports green while its target
crashes is strictly worse than one that reports red honestly. #568 documents the expectation in
tooling/gamut-fuzz/README.md instead, which is the honest minimum but does not restore the
aggregate.

The question

Should a target with a known, filed, accepted defect run somewhere that expects it to fail, so
the default-branch aggregate keeps meaning "something new is wrong"?

Options seen so far, none yet chosen:

  1. A separate job with continue-on-error: true for known-failing targets, listed by name.
    Cheap; the row still shows its result, and the job stops poisoning the aggregate. Risk: a target
    left on that list after its defect is fixed silently stops being a gate — needs its own guard
    (assert the listed targets do still fail, i.e. an expected-to-fail list, not a tolerated one).
  2. A separate workflow (fuzz-known-failing.yml) on the same schedule. Clearer separation,
    more YAML, same stale-list risk.
  3. Do nothing until fix(tiff): page_info panics on SamplesPerPixel = 0 #563 and fix(dng): raw decode sizes its buffer from declared geometry, not from the file #564 close, on the grounds that the correct fix for a red row is
    to fix the defect, and both are small. If they land quickly this issue evaporates.

Option 3 is the default if nobody picks otherwise; the point of filing is that "the aggregate is
red for a known reason" should be a recorded, revisited state rather than an ambient one.

Whatever is chosen must keep tooling/gamut-fuzz/check-targets.sh (the drift guard that
reconciles the target files, the [[bin]] entries and the matrix) able to see every target,
wherever it runs.

Refs #563, #564, #264, #568.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions