Skip to content

Make the GitHits A/B pilot ready to run: keep permissive-licence examples from GitHits' documented responses, pace example calls host-side to one per 40 s across processes and arms with the wait exclu - #153

Merged
sykosyber merged 1 commit into
mainfrom
syberlabs/e9a369f3
Oct 29, 2026

Conversation

@sykosyber

Copy link
Copy Markdown
Member

Objective: Make the GitHits A/B pilot ready to run: keep permissive-licence examples from GitHits' documented responses, pace example calls host-side to one per 40 s across processes and arms with the wait excluded from agent time, and pin the engineering smoke to Claude Code 2.1.312. The change touches the host launcher path, the GitHits licence filter, the experiment report, the smoke script, their tests and docs, and leaves githits.py's request retry behaviour alone because pilot item SW-01 measures it.

Class: R
Agent-platform: claude@opus-5.5
Admitted-by: sykosyber
Reviewed-by: rules@2 (critic check critic.rules: high 0, medium 0, low 0)
Evidence: tested

Accepted in SyberLabs thread e9a369f3-aacf-4522-82bf-2334815d2231 as candidate c1 (commit 3e2a572eeafa03cd373e4001a0b2065c1643b6e7).
Host checks on that tree: conformance passed, critic.rules passed, scenarios passed, tests passed.

…es in the licence filter (References/Sources links count as sources; answers with no code pass), pace example calls host-side to on

SyberLabs-Thread: e9a369f3-aacf-4522-82bf-2334815d2231
SyberLabs-Candidate: c1
SyberLabs-Provider: claude@opus-5.5
@sykosyber

Copy link
Copy Markdown
Member Author

Handoff: Build Thread e9a369f3 (repo-change.v5, class R, unattended, actor sykosyber) on base 90930320c6. Result 3e2a572eea, 14 files.

Evidence

  • The full suite on a scratch branch at the same change passed: https://github.com/SyberLabs/SyberWork/actions/runs/40855719696
  • Bite (the new tests run against the unchanged code) failed as intended, only on the new or changed tests: https://github.com/SyberLabs/SyberWork/actions/runs/40855724430
    • test_githits_pace fails to import because githits_pace does not exist on base.
    • test_githits.DocumentedExamples fails on the three documented responses (complete, degraded, no_example), on links under References/Sources counting as sources, and on copyleft or unknown licences still being dropped.
    • test_githits.LicensePolicy fails on licences taken from References.
    • test_githits_engineering_smoke fails on the CLI pin: '2.1.311' != '2.1.312'.
    • The result was the same on 3.11, 3.12 and Windows.
  • Conformance, run locally: matched 23 golden traces.
  • Scenarios, run locally: 11 replayed, exit 0, digest b41f7cbb2e70447c4e57bf7c253b976b81cd7ab1f95daf7b9c459564db1f44bb.
  • Thread checks on c1 passed: conformance, critic.rules, scenarios and tests.
  • run --once pass 1 found c1 acceptable. Pass 2 accepted it.
  • Certify: c1 was signed with the factory ed25519 key 4b6ae2deea. The chain head is c65d48f1d6 at 13 events.
  • Verify: signature ok, repository ok (trees, 14 changed paths and diff digest match), journal ok (13 of 13 events certified). Verdict: pass.

No paid model or real GitHits API was called. The owner merges.

🤖 Generated with Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T10:44:00.626461Z 3e2a572 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3e2a572eea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread syberlabs/githits_pace.py
Comment on lines +153 to +154
temporary.write_text(repr(now), encoding="ascii")
os.replace(temporary, stamp)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Pace from the request start, not the slot reservation

If a launcher is descheduled after writing this slot but before Client.call reaches _send, another process can send at the 40-second mark and reach GitHits less than 40 seconds after the first request—or before a delayed first request. The lock serializes reservations, not request starts, so a stalled launcher can still violate the per-organization rate; pace through request initiation or otherwise account for the actual send time.

Useful? React with 👍 / 👎.

Comment thread syberlabs/experiment.py
base=_base(events), objective=_objective(events), server=_servers(registrations, server),
models=_models(registrations), accepted=accepted, time_to_pass=time_to_pass,
reviewed=_reviewed(events, accept, commit), work=work, upto=upto, observed=_observed(events, server),
paced=_paced(events),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bound agent-time pacing to the merge

If a thread records a GitHits example after its PR has merged, this stores pacing from the entire thread, and _per_arm subtracts it from agent_time, which ends at merged_at. That wait was not part of the measured duration, so the report understates agent time (or clamps it to zero); keep the whole-thread total for paced_seconds but subtract only pacing through the merge.

Useful? React with 👍 / 👎.

@sykosyber

Copy link
Copy Markdown
Member Author

Independent verification of #153 (pilot-readiness) at 3e2a572

Verdict: pass with findings. I found nothing that blocks a merge. All the PR's claims hold. The findings below are boundaries the tests don't cover, plus one environment assumption.

Method: a read-only check of a fresh clone at head 3e2a572eeafa03cd373e4001a0b2065c1643b6e7 (base 90930320c6, 14 files). It used fakes and loopback only. It made no paid model calls and no real GitHits calls.

What was checked and holds

1. Licence filter

  • All three documented /v1/examples/search responses in the pinned fixture now pass, with results_dropped_license=0 and the body unchanged:
    • complete gives ["MIT"];
    • degraded and no_example give [].
  • Still dropped: GPL, AGPL, MPL, NOASSERTION, MIT AND GPL-2.0-only, unparsable (see repository), and a missing license.
  • Code with no source is still dropped ([None]).
  • A link to the GitHits app (githits.com / githits.dev) is skipped, and does not count as a source.
  • The three boundaries you asked about are covered in tests/test_githits.py DocumentedExamples:
    • a References link to a copyleft source: the Source 2 ... (GPL-3.0-only) case and the copyleft substitutions;
    • the section ending at the next heading: the ## Notes case, plus the thematic-break case;
    • a link on a non-code host outside the section not counting: the [docs](...) (GPL-3.0-only) case.
  • The changed assertion (a docs-only answer with no code gives [] instead of [None]) is sound:
    • With no code block and no cited source, there is nothing to license.
    • The PR adds the counterpart test: the same docs link beside a code fence still gives [None] and is dropped.
    • Its soundness depends on _CODE recognising every code form; see finding 1.
  • Dropping for SyberLabs targets and contamination is unchanged:
    • _filter is not in the diff;
    • the exclude.mentioned check still runs before the license check.

2. Pacing

  • _send and the request path are not in the diff.
  • githits.py has no sleep, retry or back-off. It gains only:
    • an optional paced_ms record field;
    • a pace= hook in run(), which syberlabs/githits_cli_entry.py:_pace supplies.
  • So nothing in the client pre-empts SW-01.
  • Only example is paced (PACED_OPS).
  • A non-loopback base is always paced at 40 s, even with SYBERLABS_GITHITS_PACE_SECONDS=0: interval_for(False) returns 40, and LauncherHook tests this.
  • The loopback exemption is documented in GAPS 63 (2).
  • A corrupt, NaN or inf timestamp means no wait. A future timestamp is capped at one interval and then overwritten (githits_pace.py:149, tested).
  • paced_ms is covered by the MAC:
    • stripping or changing it invalidates the record;
    • None, -5, "40000" and 1.5 are rejected even when re-signed;
    • the whole record is then ignored and counted in records_ignored.
  • Pacing is subtracted once:
    • time_to_pass subtracts only the pacing between the search and the evaluation (experiment.py:397);
    • the per-arm agent_time subtracts the whole thread's pacing (experiment.py:629);
    • metrics.agent_time itself is unadjusted, so nothing is subtracted twice.
  • The test arithmetic checks out:
    • 3600 − 1800 = 1800 s, which is 0.5 h;
    • the 40 s after the evaluation and the non-GitHits WebSearch paced_ms are both excluded;
    • agent_time is 2.0 − 1840/3600.

3. CLI pin

  • scripts/githits_engineering_smoke.py uses claude_code.version_tuple, which is an exact full match. It refuses:
    • 2.1.311;
    • 2.1.3120;
    • a beta suffix;
    • a banner line (tested).
  • The pin is recorded in plan.json _configuration.cli and in manifest.json cli_version.
  • No stray 2.1.311 pin remains. The only remaining mentions are a sentence in the handoff doc that explains the move, and a test case that checks 2.1.311 is refused.

4. Docs and GAPS

  • docs/GITHITS.md, docs/CLAUDE_CODE.md and docs/GITHITS_OPERATOR_HANDOFF.md match the code.
  • GAPS 63 is appended after 62 with no renumbering, and tests.test_gaps passes.

5. Local runs (Windows, C:/Python312/python.exe)

  • The 12 requested modules: 301 tests, OK. Two tests were skipped; both are pre-existing POSIX-only skips in test_githits_adversarial (needs /bin/bash; POSIX signals).
  • python -m conformance.run: matched 23 golden traces, exit 0.
  • benchmarks/run_scenarios.py: 11 scenarios replayed, exit 0, digest b41f7cbb…44bb.

6. Certificate

  • syberlabs verify checked the signature: ed25519, key 4b6ae2deea, thread e9a369f3, candidate c1, result 3e2a572eea, repo-change.v5, class R.
  • With --repo, it checked the repository: base 90930320c6 and the result are present, the trees match, and the 14 changed paths and the diff digest match.
  • Verdict: pass. The journal was not re-derived because there is no journal home. The receipt says chain_valid: true, but I did not check that independently.

Findings, ranked by severity

  1. Low. _CODE misses some code forms, so code with no source in those forms passes as [] (syberlabs/githits.py:511, :549).
    • _CODE recognises fences only at column 0–3 and indented blocks. A fenced block inside a blockquote (> ```…), or unfenced code lines, is not detected.
    • Such an answer with no cited source gives [] and passes, where main dropped it.
    • GAPS 63 (5) records only inline code as not read.
    • Suggested fix: add blockquoted fences to _CODE or to GAPS 63 (5), with one test.
  2. Low. The heading and line-ending boundaries of the References/Sources section are untested (githits.py:506–508).
    • Headings that are not recognised: singular Source / Reference, headings with extra words such as "References and further reading", and CRLF text. In CRLF text, [ \t]*$ does not match before \r.
    • Links under an unrecognised heading count only if they are on a code host.
    • The result is fail-safe when a cited source is the only one: the example is dropped as [None].
    • It is not fail-safe when there is also a recognised permissive source: a copyleft link on a non-code host under an unrecognised heading is missed. That is no worse than main, which never counted links on non-code hosts.
    • Not a regression. Worth a GAPS note or a test.
  3. Low. The default pacing directory follows the inherited TEMP/TMPDIR (githits_pace.py:75, githits_cli_entry.py:91).
    • "Across processes and arms" holds only if every launcher sees the same tempfile.gettempdir().
    • claude_code.githits_environment (claude_code.py:768) does not pin SYBERLABS_GITHITS_PACE_DIR.
    • If Claude Code, a sandbox, or a per-arm environment rewrites TMPDIR/TEMP, the arms would not share slots, and the organization's rate could be exceeded with no sign of it.
    • I did not verify whether 2.1.312 does this.
    • Suggested fix: have the provider pass an absolute host directory.
  4. Info. A call that fails or is ended during pacing writes no client record.
    • This covers pace_failed and a SIGTERM while queued; pacing runs before Client.call, at githits.py:1384.
    • The host flags such a call as unrecorded. This is documented in GAPS 63 (3) and in the launcher docstring.
    • Also, an example sent with no token still waits for its slot before it fails as no_token. This is harmless.
  5. Info. The per-arm agent_time subtracts the whole thread's pacing, including any after merged_at. This is bounded by max(0, …) and in practice is zero.

Interplay with #152

CI

gh pr checks 153: 25 of 25 pass, 0 pending, 0 failing, at head 3e2a572 (re-checked just before posting).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant