Skip to content

Let the bot take result files, and add a benchmark-processor Copilot agent (1.12.0.0) - #40

Merged
MattyStacks merged 3 commits into
mainfrom
feat/issue-attachments
Oct 26, 2026
Merged

MattyStacks merged 3 commits into
mainfrom
feat/issue-attachments

Conversation

@MattyStacks

@MattyStacks MattyStacks commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Pulls the backlog item "Let the bot process attached or pasted result files" (prompted by #36).

What changes

  • Result files through issues. A new Benchmark result (file) form takes one attached or pasted .yaml/.yml/.json result file. Any issue with the benchmark label and one result file (like Add my benchmark run to the benchmarks results #36's Open GPU Benchmarker export) works too.
    • Downloads come only from github.com/user-attachments/files/, with redirects allowed only to *.githubusercontent.com, capped at 1 MB and 20 s, and saved into $OUT, never the checkout.
    • YAML aliases are refused. The file's fields are kept (including detected and diagnostics).
    • submitted_by comes only from the Credit box. testbench_id, source, and synthetic are dropped.
  • Submitting is consent. Both forms drop their required CC BY 4.0 and frame-generation checkboxes and show the license as a notice. The PR body says the result was submitted under CC BY 4.0, and the reviewer checklist adds a frame-generation item.
  • Fallbacks.
    • An issue with neither the form nor a file gets a reply linking both forms.
    • Attachments that aren't result files (CSVs, screenshots) exit 3. The bot replies and adds needs-maintainer, and the steps a maintainer follows are in the new docs/PROCESSING_ISSUES.md.
  • benchmark-processor Copilot agent (.github/agents/benchmark-processor.agent.md). Assign a benchmark issue to Copilot and pick this agent.
    • It writes the result YAML itself from whatever the issue holds: pasted numbers, a plain description, a file, a CSV or Cyberpunk capture (through parse.py / import_cyberpunk.py), or a screenshot.
    • It adds a missing GPU or game with sources, runs validate.py / build.py / test_build.py, and opens a data-only PR listing every assumption. If something can't be worked out, it opens a draft with questions instead.
    • It may only change data/, and it treats the issue and attachments as untrusted.
    • copilot-setup-steps.yml (one read-only job, no secrets) installs its Python dependencies. parse.py gains --date.
  • Converting raw captures is still in the backlog, along with the openbenchmarker-repo export change.

Testing

  • New check_issue_files() covers:
    • an attached file, a pasted file in the form, and a pasted JSON block;
    • a CSV attachment going to a maintainer;
    • nine kinds of failure;
    • the redirect allowlist.
  • It uses the hidden --attachment-dir and never touches the network. check_bot_safety() makes sure the workflow never passes that flag.
  • Ran the script live against Add my benchmark run to the benchmarks results #36's real attachment on a temp copy of data/. It wrote data/community/rtx_5080_16gb_fe/result_20261026_openbenchmarker_neon_flythrough.yaml, which passes the checks with two warnings (no proof, no official baseline).
  • validate.py, build.py, test_build.py, py_compile, the blog build, and git diff --check all pass.

After merge

  • Run "Copilot Setup Steps" once from Actions, then try the agent on a test issue (it only works from main). If downloads are blocked, add objects.githubusercontent.com to Settings → Copilot → Internet access.
  • Tag v1.12.0.0.
  • Edit or relabel Add my benchmark run to the benchmarks results #36 so the bot reruns.

🤖 Generated with Claude Code

A new "Benchmark result (file)" form, and any issue with the benchmark label,
can carry one .yaml/.yml/.json result file, attached or pasted. The bot
downloads only from github.com/user-attachments/files/ (redirects only to
*.githubusercontent.com, 1 MB and 20 s caps), refuses YAML aliases, keeps the
file's fields, sets submitted_by only from the Credit box, and drops
testbench_id, source, and synthetic. Attachments it can't convert exit 3 and
get the needs-maintainer label (docs/PROCESSING_ISSUES.md).

Submitting the issue is now the CC BY 4.0 agreement: both forms drop their
required checkboxes and show the license as a notice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Cloudflare Pages preview

MattyStacks and others added 2 commits October 26, 2026 18:32
Assign a benchmark issue to Copilot and pick benchmark-processor: it writes
the result YAML from whatever the issue holds (pasted numbers, a file, a raw
capture, a screenshot), adds a missing GPU or game with sources, runs the
checks, and opens a data-only pull request ready to approve, or a draft with
questions. copilot-setup-steps.yml installs the Python dependencies, and
parse.py takes --date so captures get the run's day.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MattyStacks MattyStacks changed the title Let the benchmark bot take attached or pasted result files (1.12.0.0) Let the bot take result files, and add a benchmark-processor Copilot agent (1.12.0.0) Oct 26, 2026
@MattyStacks
MattyStacks merged commit 18f513b into main Oct 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant