Skip to content

Add Markdown summaries of review outcomes for CI job pages - #41

Open
iamkhalidd wants to merge 1 commit into
prooflinedev:mainfrom
iamkhalidd:review-outcome-step-summary
Open

iamkhalidd wants to merge 1 commit into
prooflinedev:mainfrom
iamkhalidd:review-outcome-step-summary

Conversation

@iamkhalidd

Copy link
Copy Markdown

Who this helps

Teams that call wait_for_review in CI. Today the job either passes or fails, and to see why a review was incomplete they have to print the ReviewOutcome themselves or open Proofline. This PR puts the answer on the GitHub Actions run page.

What it adds

  • outcome_markdown(outcome) -> str renders a ReviewOutcome as Markdown: status, a link to the review, the finding count, one row per assessment gap with its reason codes, and one row per unresolved check (name, outcome, lifecycle).
  • write_github_step_summary(outcome, path=None) -> bool appends that Markdown to GITHUB_STEP_SUMMARY. Outside Actions (variable unset) it does nothing and returns False, so the same script runs locally.
  • A README section with a short example.

Decisions

  • Never looks like a pass by mistake. Every status other than COMPLETED (INCOMPLETE, FAILED, STOPPED, TIMED_OUT) says "Do not treat this result as a pass."
  • No free text. CheckGap.detail and unresolved_reason are left out because they can carry withheld environment values. The summary links to the review instead. Job summaries are visible to anyone who can read the repo, so this felt like the safe default.
  • Server text is escaped. Check names, statuses and reason codes are backslash-escaped, newlines are flattened and cells are capped at 200 characters, so they can't break the table or add links or HTML. The review URL is only linked when it's a plain http(s) URL.
  • Bounded size. Each table stops at 50 rows and says how many more are in the review.
  • Pure functions, no client changes. It works the same for ProoflineClient and AsyncProoflineClient.

Verification

  • tests/test_summary.py (13 tests): all statuses, gap and check rows, detail/unresolved_reason never shown, hostile check names (pipes, newlines, <img>, links, backticks), unsafe URL not linked, row cap, append to an existing summary, no-op outside Actions, explicit path.
  • Full suite: 149 passed. ruff check, ruff format --check and mypy proofline pass.
  • I couldn't run scripts/check.py on Windows, because the pinned uv 0.10.12 bootstrap has no Windows checksum, so I ran the same tools from requirements-dev.txt in a venv. CI will run the full gate.

Example output:

### Proofline review incomplete

The review ended without the evidence needed to call it complete. Do not treat this result as a pass.

[Open the review in Proofline](https://proofline.sh/...)

| Assessment | Status | Reasons |
| --- | --- | --- |
| `1234abcd` | COMPLETED | CHECKS\_UNRESOLVED |

| Assessment | Check | Outcome | Lifecycle |
| --- | --- | --- | --- |
| `1234abcd` | unit tests | FAILING | GATING |

outcome_markdown renders a ReviewOutcome as Markdown and
write_github_step_summary appends it to GITHUB_STEP_SUMMARY. Statuses
other than COMPLETED are marked as not a pass, free-text check detail is
left out, and server text is escaped.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant