Repository navigation
feat(oracles): run pinned upstream suites in test:oracles - #31
developerz-ai[bot] merged 2 commits into
Conversation
Replace the abort at apps/fast-rspec/Rakefile:17-18 with a real oracle runner: it discovers test/oracles/<gem>/ dirs, fetches the pinned upstream commit (tag fallback) into tmp/oracles, bundle-installs the overlay Gemfile, executes each run script with lib/compat unshifted onto $LOAD_PATH, and classifies every suite as pass / skip(reason) / fail. Skips (unfetched checkout, install failure, crash before results, timeout) are reported with their reason and never gate; an un-skipped failure gates with exit 1. Rows print per suite (pin, counts, %) and land in tmp/oracles/results.json for docs/compat.md recording. 10 pinned oracles added: minitest 5.27.0, rspec-core v3.13.2, rspec-expectations v3.13.3, rspec-mocks v3.13.2, webmock (master head, no release tags), vcr v6.4.0, faker v3.8.0, database_cleaner v2.1.0, simplecov v0.22.0, factory_bot v6.6.0. Frontends are unimplemented today, so suites classify skip(did-not-run) — the honest leg-1 output the compat table consumes. Offline bootstrap unit tests cover discovery, ORACLE= filter, the classification matrix, timeouts, skip.txt and exit codes via fake local-pin oracle dirs; no network or git in the unit suite. AGENTS.md commands table updated to the implemented behavior. Dz-Task-Id: tsk_b5b49c53d57ce7796cd234dadce53ed9
|
Implement the oracle suite execution path
Work landed on #31 — watching it toward merge. 3h46m · 474 tool calls · 134 messages 🤖 developerz.ai, an automated agent, not a human — developerz-ai[bot] |
🟡 Reviewed9 actionable comment(s) · 2 refuted by verification · 1 suppressed (anti-noise) · grounded on your code 🤖 developerz.ai — automated review, running on your model and your box. |
|
Waiting for a review of this head. Nothing is needed from you right now.
Status for head 🤖 Posted by developerz.ai — an automated agent, not a human. |
There was a problem hiding this comment.
Review summary — 45 file(s), 15 finding(s).
Critical 0 · Major 4 · Minor 11 · Nit 0
Adds pinned upstream RSpec oracle suites (rspec-core/expectations/mocks: .sha, overlay Gemfile, run script, skip.txt). The overlay Gemfiles point the fast_rspec path gem one directory short, so every install fails and every suite lands on skip — no parity coverage yet.
| File | Findings |
|---|---|
apps/fast-rspec/test/oracles/database_cleaner/Gemfile |
1 major |
apps/fast-rspec/test/oracles/minitest/run |
1 minor |
apps/fast-rspec/test/oracles/rspec-core/.sha |
1 minor |
apps/fast-rspec/test/oracles/rspec-core/Gemfile |
1 major |
apps/fast-rspec/test/oracles/rspec-core/run |
1 minor |
apps/fast-rspec/test/oracles/support/compat_shadow.rb |
1 major |
apps/fast-rspec/test/oracles/support/runner.rb |
1 major, 2 minor |
apps/fast-rspec/test/unit/oracles_runner_test.rb |
6 minor |
- some test files were not shown to the reviewer — they were read at the reviewed commit, but did not fit the tests section; the least-changed ones were left out
🤖 developerz.ai review — automated, running on your model and your box. What is this?
|
File fence widened for a findings fix. The
Why: Thread PRRT_kwDOVeg6bs62a_9_ asks to extract the suite-totals -> ORACLE RESULT parsing out of the ten bash run scripts into "one shared helper under test/oracles/support" and to assert it in the unit suite. The shared helper is a new file next to support/runner.rb and support/compat_shadow.rb; without it the parsing regexes stay untested in bash only. The platform granted it under the same area lock a new task takes (no other in-flight task holds these files), after checking the path is not a secret, CI configuration or a human-merge path. Posted automatically by the developerz bot (AI-authored). |
…mfile (+27 more) Follow-up intent: Address review findings on #31 Dz-Task-Id: tsk_0565342e341cda0412bae391d836b5ad
There was a problem hiding this comment.
Review summary — 28 file(s), 9 finding(s).
Critical 0 · Major 0 · Minor 9 · Nit 0
Adds the oracle runner (fetch, bundle install, run, classify), a shared totals parser and the compat shadow, with offline unit coverage. Risk: the rspec oracles now fetch the monorepo while their pins/overlays still target the split repos, and a pass row proves nothing about fast_rspec.
| File | Findings |
|---|---|
apps/fast-rspec/test/oracles/minitest/run |
1 minor |
apps/fast-rspec/test/oracles/rspec-core/Gemfile |
1 minor |
apps/fast-rspec/test/oracles/support/compat_shadow.rb |
2 minor |
apps/fast-rspec/test/oracles/support/runner.rb |
3 minor |
apps/fast-rspec/test/oracles/support/totals.rb |
1 minor |
apps/fast-rspec/test/unit/oracles_runner_test.rb |
1 minor |
9 non-blocking finding(s) below major (no thread on a re-review — only blocking findings open threads)
apps/fast-rspec/test/oracles/minitest/run:31minor · docs The header at line 5 says "run scripts never touch RUBYOPT or the bundle themselves", but this change addsRUBYOPT=here, so the file now contradicts itself about env handling. Update line 5 to state the one exception: RUBYOPT is cleared for the plain-Ruby totals helper (per the note at lines 29-…apps/fast-rspec/test/oracles/rspec-core/Gemfile:5minor · buggem 'aruba'is declared only here but the suite runsbundle exec rspecin the rspec-core directory of the upstream monorepo, whose specs shell out through aruba; without it in the bundle the specs fail to load and the oracle silently degrades to a skip. Keep the rspec-expectations and rspec-moc…apps/fast-rspec/test/oracles/support/compat_shadow.rb:11minor · test No test executes this shim — the offline suite stubs run scripts as /bin/sh — so removingrequire 'bundler/setup'here would LoadError everybundle execoracle run into skip(crash) while all unit tests stay green. Pin the order offline: spawn ruby with RUBYOPT pointing at this shim against a tm…apps/fast-rspec/test/oracles/support/compat_shadow.rb:15minor · test The shim requires fast_rspec but leaves no trace in the suite output, so the runner cannot tell whether the suite actually ran the shadowed code — a shadow that does not take effect still yields a 'pass' row (the unit test's own PASS_SCRIPT, which contains no fast_rspec at all, is already classifie…apps/fast-rspec/test/oracles/support/runner.rb:194minor · bug totals.rb can legally print a negative pass count (rspec load errors exceeding passing examples, e.g. '3 examples, 0 failures, 5 errors occurred outside of examples' → pass=-2), which thispass=(\d+)regex rejects, so the runner classifies that suite as skip('no ORACLE RESULT after exit 0') inste…apps/fast-rspec/test/oracles/support/runner.rb:264minor · bug skip.txt is only ever looked up with the suite-level id '<oracle>:<stage>', but the skip.txt files this PR adds document per-example known-failing ids, so a documented failure never matches and the suite still ends 'fail' with exit 1. Either look the failing example ids up or make the ids stage-lev…apps/fast-rspec/test/oracles/support/runner.rb:349minor · bug One suite that cannot be spawned at all (no git, bundle or bash on PATH: Errno::ENOENT from spawn) raises out of the runner: no table, no tmp/oracles/results.json, and the remaining oracles never run. Wrap the per-suite call so such a suite classifies as skip with its reason.apps/fast-rspec/test/oracles/support/totals.rb:107minor · bug The helper reads the suite log as UTF-8, so a suite that prints a byte that is not valid UTF-8 makes the regexp match raise ArgumentError (invalid byte sequence in UTF-8); the run script then exits without an ORACLE RESULT line and the oracle is classified as skip, silently losing a suite that actu…apps/fast-rspec/test/unit/oracles_runner_test.rb:704minor · test The fake run scripts print the RESULT line by hand (oracles_runner_test.rb:22), so nothing pins the shell→totals→runner contract this helper creates: a run script that calls totals.rb with the wrong --dialect prints no totals line and the runner classifies that oracle as skip. Add one oracle whose…
- some test files were not shown to the reviewer — they were read at the reviewed commit, but did not fit the tests section; the least-changed ones were left out
🤖 developerz.ai review — automated, running on your model and your box. What is this?
What
Replace the abort at apps/fast-rspec/Rakefile:17-18 with a real oracle
runner: it discovers test/oracles// dirs, fetches the pinned
upstream commit (tag fallback) into tmp/oracles, bundle-installs the
overlay Gemfile, executes each run script with lib/compat unshifted onto
$LOAD_PATH, and classifies every suite as pass / skip(reason) / fail.
Skips (unfetched checkout, install failure, crash before results,
timeout) are reported with their reason and never gate; an un-skipped
failure gates with exit 1. Rows print per suite (pin, counts, %) and
land in tmp/oracles/results.json for docs/compat.md recording.
10 pinned oracles added: minitest 5.27.0, rspec-core v3.13.2,
rspec-expectations v3.13.3, rspec-mocks v3.13.2, webmock (master head,
no release tags), vcr v6.4.0, faker v3.8.0, database_cleaner v2.1.0,
simplecov v0.22.0, factory_bot v6.6.0. Frontends are unimplemented
today, so suites classify skip(did-not-run) — the honest leg-1 output
the compat table consumes.
Offline bootstrap unit tests cover discovery, ORACLE= filter, the
classification matrix, timeouts, skip.txt and exit codes via fake
local-pin oracle dirs; no network or git in the unit suite. AGENTS.md
commands table updated to the implemented behavior.
Acceptance
Changes
apps/fast-rspec/test/oracles/support/runner.rb(create) — New dev tooling, not shipped (gemspec stays untouched). Defines Oracles::Runner: discovers test/oracles/*/ dirs under the app root, honors ORACLE env filter; per suite reads .sha (line 1 pin, line 2 tag fallback, 'local ' bypasses git), git-fetches the pin into /tmp/oracles cache (skip with reason on failure), bundle-installs the overlay Gemfile (skip with reason on failure), executes run with ORACLE_SRC/FAST_RSPEC_ROOT/ORACLE_NAME env and RUBYOPT prefixed with '-I<support_dir> -rcompat_shadow', under Process.spawn with ORACLE_TIMEOUT (default 1800s). Classifies: ORACLE R [trimmed]apps/fast-rspec/test/oracles/support/compat_shadow.rb(create) — RUBYOPT-loaded shim the runner injects into oracle suites: unshifts apps/fast-rspec/lib/compat onto the front of $LOAD_PATH (after bundler/setup has installed gem paths, so compat require paths shadow upstream rspec/minitest) and requires 'fast_rspec'. ~15 lines, frozen literal, no comments beyond the one non-obvious reason (why unshift happens at require time).apps/fast-rspec/test/unit/oracles_runner_test.rb(create) — Bootstrap-harness unit test (require_relative '../bootstrap/harness', same DSL as test/unit/check_test.rb) named to match the test/unit/**/*_test.rb glob. Fully offline: builds fake oracle dirs in Dir.mktmpdir with 'local ' pins and stub run scripts to cover: dir discovery + ORACLE filter; pin/tag/local .sha parsing; classification matrix (RESULT line exit 0 => pass; RESULT with failures => fail; crash without RESULT => skip with reason; ORACLE_TIMEOUT expiry => skip; local pin bypasses git); skip.txt id merge; summary row shape (name, pin, counts, %, reason) and tmp/oracles/res [trimmed]apps/fast-rspec/Rakefile(edit) — Replace the abort at lines 17-18: task :oracles in the test namespace requires_relative 'test/oracles/support/runner' and exits with Oracles::Runner.new(dir).run(ENV['ORACLE']). Keep the desc (mention ORACLE= for one oracle). Everything else in the file stays byte-identical.AGENTS.md(edit) — Commands table must stay aligned with the Rakefile (its own rule): line 44 Full gate row drops 'oracle stage currently fails as unimplemented' and says the oracle stage runs the pinned upstream suites where skips-with-reason don't gate; line 50 row 'Oracles (unavailable)' becomes 'Oracles | bin/rake test:oracles (ORACLE= for one): fetches the pinned upstream suite per test/oracles//.sha, runs it with lib/compat shadowed, prints per-suite pin/pass/skip/% rows; skips never gate'. Also line 86 testing-rules bullet 'oracle execution is not implemented yet' drops that clause. N [trimmed]apps/fast-rspec/test/oracles/minitest/.sha(create) — Pin for the minitest oracle: line 1 = 819e9b5f873d7ab6d4edfeaed48542279367ff93 (v5.27.0 of minitest/minitest), line 2 = 'tag v5.27.0' (fallback fetch ref), '# '-comments allowed. Runner reads line 1 as the pin it prints and fetches by sha, falling back to the tag.apps/fast-rspec/test/oracles/minitest/Gemfile(create) — Overlay bundling for the minitest oracle: frozen_string_literal comment, source rubygems, gem 'fast_rspec', path: File.expand_path('../..', dir) and gem 'minitest', path: ENV.fetch('ORACLE_SRC'), plus gem 'rake' (suite is rake-based). Bundler writes Gemfile.lock beside it (covered by root .gitignore).apps/fast-rspec/test/oracles/minitest/run(create) — Suite script for the minitest oracle: '#!/usr/bin/env bash', 'set -euo pipefail', 'cd "$ORACLE_SRC"', 'bundle exec rake'. No RUBYOPT handling (the runner injects compat shadowing via env). When minitest runs to completion it should print ORACLE RESULT pass= fail= skip=; today it crashes at load and the runner classifies that as skip with the reason.apps/fast-rspec/test/oracles/minitest/skip.txt(create) — Known-failing example ids for the minitest oracle: header comment documenting the '# ()' format and that entries only shrink; zero entries today (every failure is a real defect once suites run).apps/fast-rspec/test/oracles/rspec-core/.sha(create) — Pin for the rspec-core oracle: line 1 = 3bdf310e172836d28da49349f00e4747e6c29b05 (v3.13.2 of rspec/rspec-core — newest real tag; docs/target's 3.13.6 has no upstream ref), line 2 = 'tag v3.13.2'.apps/fast-rspec/test/oracles/rspec-core/Gemfile(create) — Overlay bundling for the rspec-core oracle: fast_rspec path ../.., gem 'rspec-core', path: ENV.fetch('ORACLE_SRC'), plus gem 'rake'. Bundler resolves rspec-core's runtime deps (rspec-support etc.) from rubygems.apps/fast-rspec/test/oracles/rspec-core/run(create) — Suite script for rspec-core: cd "$ORACLE_SRC" && bundle exec rspec (the spec/ suite, not cucumber features). Emits ORACLE RESULT line once the suite completes; load crash today => skip with reason.apps/fast-rspec/test/oracles/rspec-core/skip.txt(create) — Known-failing ids for rspec-core oracle: documented header, zero entries today.apps/fast-rspec/test/oracles/rspec-expectations/.sha(create) — Pin for rspec-expectations oracle: line 1 = 555979f5bae76e83a52f166dd7060daf504c6177 (v3.13.3 of rspec/rspec-expectations — newest real tag), line 2 = 'tag v3.13.3'.apps/fast-rspec/test/oracles/rspec-expectations/Gemfile(create) — Overlay bundling: fast_rspec path ../.., gem 'rspec-expectations', path: ENV.fetch('ORACLE_SRC'), plus gem 'rake'.apps/fast-rspec/test/oracles/rspec-expectations/run(create) — Suite script: cd "$ORACLE_SRC" && bundle exec rspec. Emits ORACLE RESULT line on completion.apps/fast-rspec/test/oracles/rspec-expectations/skip.txt(create) — Known-failing ids: documented header, zero entries today.apps/fast-rspec/test/oracles/rspec-mocks/.sha(create) — Pin for rspec-mocks oracle: line 1 = 0d6a02313cd5b3cc6f58edcd8b1a8f019cb6a4a0 (v3.13.2 of rspec/rspec-mocks — newest real tag), line 2 = 'tag v3.13.2'.apps/fast-rspec/test/oracles/rspec-mocks/Gemfile(create) — Overlay bundling: fast_rspec path ../.., gem 'rspec-mocks', path: ENV.fetch('ORACLE_SRC'), plus gem 'rake'.apps/fast-rspec/test/oracles/rspec-mocks/run(create) — Suite script: cd "$ORACLE_SRC" && bundle exec rspec. Emits ORACLE RESULT line on completion.apps/fast-rspec/test/oracles/rspec-mocks/skip.txt(create) — Known-failing ids: documented header, zero entries today.apps/fast-rspec/test/oracles/webmock/.sha(create) — Pin for webmock oracle: line 1 = 4273387030f7a3f498f9e4ac5f90f5dde8f68086 (webmock/webmock master head — the repo carries NO release tags in this environment, rubygems latest is 3.34.0; note that in a '# ' comment so the movable pin is visible), no tag line.apps/fast-rspec/test/oracles/webmock/Gemfile(create) — Overlay bundling: fast_rspec path ../.., gem 'webmock', path: ENV.fetch('ORACLE_SRC'), plus gem 'rake' (suite is rake-based).apps/fast-rspec/test/oracles/webmock/run(create) — Suite script: cd "$ORACLE_SRC" && bundle exec rake (webmock's default minitest task). Emits ORACLE RESULT line on completion.apps/fast-rspec/test/oracles/webmock/skip.txt(create) — Known-failing ids: documented header, zero entries today.Verification
./bin/checkpassed (exit 0) in 47s.What the editors ran to see the change working, and what they saw:
cd apps/fast-rspec && for f in test/oracles/*/run; do bash -n "$f" || exit 1; done && bundle exec rubocop test/oracles/*/Gemfile(pins): exit 0; rubocop: "10 files inspected, no offenses detected" — every run script parses, every overlay Gemfile lint-clean (frozen literals, ordered gems, TargetRubyVersion 3.2)chmod 0755 apps/fast-rspec/test/oracles/*/run && ls -l apps/fast-rspec/test/oracles/*/run(pins): all ten run scripts listed as -rwxr-xr-x — required because the runner's checked_pin aborts unless run is File.executable? and spawns it directlycode_grep '^[0-9a-f]{40}$' and '^tag .+' over test/oracles/*.sha(pins): 10 pins on line 1 exactly matching the brief's table (minitest 819e9b5..., rspec-core 3bdf310..., rspec-expectations 555979f..., rspec-mocks 0d6a023..., webmock 4273387..., vcr 5dcacdf..., faker e48d35f..., database_cleaner 23fb7a7..., simplecov 0e35b25..., factory_bot 5d3995...); 9 tag fallback lines, webmock correctly taglesscode_web_fetch api.github.com/repos/minitest/minitest/commits/v5.27.0(pins): the v5.27.0 tag resolves to exactly 819e9b5f873d7ab6d4edfeaed48542279367ff93 ("Branching minitest to version 5.27.0"), so the tag fallback ref in minitest/.sha is realcd apps/fast-rspec && bundle exec rake test(runner): 44 tests, 0 failurescd apps/fast-rspec && bundle exec rake -T test:oracles(runner): rake test:oracles # Run upstream suites (rspec-core, minitest, factory_bot, …) against FastRspec (ORACLE= for one)cd apps/fast-rspec && bundle exec rubocop test/oracles/support test/unit/oracles_runner_test.rb Rakefile(runner): 4 files inspected, 13 offenses — 5 in test/oracles/support/runner.rb (Metrics/AbcSize 17.35/17 + 4 correctable), 8 in test/unit/oracles_runner_test.rb (2 Met…(truncated)
Part 1/2 of a goal
🤖 Opened by developerz.ai, an automated agent, not a human, on developerz-ai/fast_rspec.
Raised automatically by a developerz.ai maintainer lane.