Skip to content

feat(oracles): run pinned upstream suites in test:oracles - #31

Merged
developerz-ai[bot] merged 2 commits into
mainfrom
dz/implement-the-oracle-suite-execution-path--b5b49c53
Nov 12, 2026
Merged

developerz-ai[bot] merged 2 commits into
mainfrom
dz/implement-the-oracle-suite-execution-path--b5b49c53

Conversation

@developerz-ai

@developerz-ai developerz-ai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

What

Replace the abort at apps/fast-rspec/Rakefile:17-18 with a real oracle
runner: it discovers test/oracles// dirs, fetches the pinned
upstream commit (tag fallback) into tmp/oracles, bundle-installs the
overlay Gemfile, executes each run script with lib/compat unshifted onto
$LOAD_PATH, and classifies every suite as pass / skip(reason) / fail.
Skips (unfetched checkout, install failure, crash before results,
timeout) are reported with their reason and never gate; an un-skipped
failure gates with exit 1. Rows print per suite (pin, counts, %) and
land in tmp/oracles/results.json for docs/compat.md recording.

10 pinned oracles added: minitest 5.27.0, rspec-core v3.13.2,
rspec-expectations v3.13.3, rspec-mocks v3.13.2, webmock (master head,
no release tags), vcr v6.4.0, faker v3.8.0, database_cleaner v2.1.0,
simplecov v0.22.0, factory_bot v6.6.0. Frontends are unimplemented
today, so suites classify skip(did-not-run) — the honest leg-1 output
the compat table consumes.

Offline bootstrap unit tests cover discovery, ORACLE= filter, the
classification matrix, timeouts, skip.txt and exit codes via fake
local-pin oracle dirs; no network or git in the unit suite. AGENTS.md
commands table updated to the implemented behavior.

Acceptance

  • bin/rake test:oracles no longer aborts: it discovers test/oracles/*/ dirs, fetches each .sha pin (tag fallback), bundle-installs the overlay and executes run — verified by offline unit tests using fake local: pin dirs
  • Each oracle run emits one per-suite row (name, pin sha, pass/fail/skip counts, %, verdict + reason) and writes the same rows to tmp/oracles/results.json, mirroring docs/compat.md columns (Pinned/Pass/Skip/%)
  • Skips never gate: checkout failure, bundle install failure, suite crash before ORACLE RESULT, and ORACLE_TIMEOUT each classify as skip with the reason printed; exit code is 1 only when a suite ran and failed un-skipped
  • ORACLE= env runs exactly one oracle; without it all 10 run
  • bundle exec rake test passes with the new test/unit/oracles_runner_test.rb covering discovery, filter, pin/tag/local pin reading, the classification matrix, skip.txt parsing, timeout, summary table and exit codes — no network, no git
  • bundle exec rubocop is clean on Rakefile, test/oracles/support/*.rb, test/unit/oracles_runner_test.rb and all 10 overlay Gemfiles (frozen literals, target Ruby 3.2)
  • All 10 .sha files carry the real upstream pins fetched this run: minitest v5.27.0 819e9b5..., rspec-core v3.13.2, rspec-expectations v3.13.3, rspec-mocks v3.13.2, webmock master 4273387..., vcr v6.4.0, faker v3.8.0, database_cleaner v2.1.0, simplecov v0.22.0, factory_bot v6.6.0
  • AGENTS.md commands table matches implemented behavior: Full gate row no longer says the oracle stage fails as unimplemented; Oracles row documents the real task, ORACLE= filter and per-suite output

Changes

  • apps/fast-rspec/test/oracles/support/runner.rb (create) — New dev tooling, not shipped (gemspec stays untouched). Defines Oracles::Runner: discovers test/oracles/*/ dirs under the app root, honors ORACLE env filter; per suite reads .sha (line 1 pin, line 2 tag fallback, 'local ' bypasses git), git-fetches the pin into /tmp/oracles cache (skip with reason on failure), bundle-installs the overlay Gemfile (skip with reason on failure), executes run with ORACLE_SRC/FAST_RSPEC_ROOT/ORACLE_NAME env and RUBYOPT prefixed with '-I<support_dir> -rcompat_shadow', under Process.spawn with ORACLE_TIMEOUT (default 1800s). Classifies: ORACLE R [trimmed]
  • apps/fast-rspec/test/oracles/support/compat_shadow.rb (create) — RUBYOPT-loaded shim the runner injects into oracle suites: unshifts apps/fast-rspec/lib/compat onto the front of $LOAD_PATH (after bundler/setup has installed gem paths, so compat require paths shadow upstream rspec/minitest) and requires 'fast_rspec'. ~15 lines, frozen literal, no comments beyond the one non-obvious reason (why unshift happens at require time).
  • apps/fast-rspec/test/unit/oracles_runner_test.rb (create) — Bootstrap-harness unit test (require_relative '../bootstrap/harness', same DSL as test/unit/check_test.rb) named to match the test/unit/**/*_test.rb glob. Fully offline: builds fake oracle dirs in Dir.mktmpdir with 'local ' pins and stub run scripts to cover: dir discovery + ORACLE filter; pin/tag/local .sha parsing; classification matrix (RESULT line exit 0 => pass; RESULT with failures => fail; crash without RESULT => skip with reason; ORACLE_TIMEOUT expiry => skip; local pin bypasses git); skip.txt id merge; summary row shape (name, pin, counts, %, reason) and tmp/oracles/res [trimmed]
  • apps/fast-rspec/Rakefile (edit) — Replace the abort at lines 17-18: task :oracles in the test namespace requires_relative 'test/oracles/support/runner' and exits with Oracles::Runner.new(dir).run(ENV['ORACLE']). Keep the desc (mention ORACLE= for one oracle). Everything else in the file stays byte-identical.
  • AGENTS.md (edit) — Commands table must stay aligned with the Rakefile (its own rule): line 44 Full gate row drops 'oracle stage currently fails as unimplemented' and says the oracle stage runs the pinned upstream suites where skips-with-reason don't gate; line 50 row 'Oracles (unavailable)' becomes 'Oracles | bin/rake test:oracles (ORACLE= for one): fetches the pinned upstream suite per test/oracles//.sha, runs it with lib/compat shadowed, prints per-suite pin/pass/skip/% rows; skips never gate'. Also line 86 testing-rules bullet 'oracle execution is not implemented yet' drops that clause. N [trimmed]
  • apps/fast-rspec/test/oracles/minitest/.sha (create) — Pin for the minitest oracle: line 1 = 819e9b5f873d7ab6d4edfeaed48542279367ff93 (v5.27.0 of minitest/minitest), line 2 = 'tag v5.27.0' (fallback fetch ref), '# '-comments allowed. Runner reads line 1 as the pin it prints and fetches by sha, falling back to the tag.
  • apps/fast-rspec/test/oracles/minitest/Gemfile (create) — Overlay bundling for the minitest oracle: frozen_string_literal comment, source rubygems, gem 'fast_rspec', path: File.expand_path('../..', dir) and gem 'minitest', path: ENV.fetch('ORACLE_SRC'), plus gem 'rake' (suite is rake-based). Bundler writes Gemfile.lock beside it (covered by root .gitignore).
  • apps/fast-rspec/test/oracles/minitest/run (create) — Suite script for the minitest oracle: '#!/usr/bin/env bash', 'set -euo pipefail', 'cd "$ORACLE_SRC"', 'bundle exec rake'. No RUBYOPT handling (the runner injects compat shadowing via env). When minitest runs to completion it should print ORACLE RESULT pass= fail= skip=; today it crashes at load and the runner classifies that as skip with the reason.
  • apps/fast-rspec/test/oracles/minitest/skip.txt (create) — Known-failing example ids for the minitest oracle: header comment documenting the '# ()' format and that entries only shrink; zero entries today (every failure is a real defect once suites run).
  • apps/fast-rspec/test/oracles/rspec-core/.sha (create) — Pin for the rspec-core oracle: line 1 = 3bdf310e172836d28da49349f00e4747e6c29b05 (v3.13.2 of rspec/rspec-core — newest real tag; docs/target's 3.13.6 has no upstream ref), line 2 = 'tag v3.13.2'.
  • apps/fast-rspec/test/oracles/rspec-core/Gemfile (create) — Overlay bundling for the rspec-core oracle: fast_rspec path ../.., gem 'rspec-core', path: ENV.fetch('ORACLE_SRC'), plus gem 'rake'. Bundler resolves rspec-core's runtime deps (rspec-support etc.) from rubygems.
  • apps/fast-rspec/test/oracles/rspec-core/run (create) — Suite script for rspec-core: cd "$ORACLE_SRC" && bundle exec rspec (the spec/ suite, not cucumber features). Emits ORACLE RESULT line once the suite completes; load crash today => skip with reason.
  • apps/fast-rspec/test/oracles/rspec-core/skip.txt (create) — Known-failing ids for rspec-core oracle: documented header, zero entries today.
  • apps/fast-rspec/test/oracles/rspec-expectations/.sha (create) — Pin for rspec-expectations oracle: line 1 = 555979f5bae76e83a52f166dd7060daf504c6177 (v3.13.3 of rspec/rspec-expectations — newest real tag), line 2 = 'tag v3.13.3'.
  • apps/fast-rspec/test/oracles/rspec-expectations/Gemfile (create) — Overlay bundling: fast_rspec path ../.., gem 'rspec-expectations', path: ENV.fetch('ORACLE_SRC'), plus gem 'rake'.
  • apps/fast-rspec/test/oracles/rspec-expectations/run (create) — Suite script: cd "$ORACLE_SRC" && bundle exec rspec. Emits ORACLE RESULT line on completion.
  • apps/fast-rspec/test/oracles/rspec-expectations/skip.txt (create) — Known-failing ids: documented header, zero entries today.
  • apps/fast-rspec/test/oracles/rspec-mocks/.sha (create) — Pin for rspec-mocks oracle: line 1 = 0d6a02313cd5b3cc6f58edcd8b1a8f019cb6a4a0 (v3.13.2 of rspec/rspec-mocks — newest real tag), line 2 = 'tag v3.13.2'.
  • apps/fast-rspec/test/oracles/rspec-mocks/Gemfile (create) — Overlay bundling: fast_rspec path ../.., gem 'rspec-mocks', path: ENV.fetch('ORACLE_SRC'), plus gem 'rake'.
  • apps/fast-rspec/test/oracles/rspec-mocks/run (create) — Suite script: cd "$ORACLE_SRC" && bundle exec rspec. Emits ORACLE RESULT line on completion.
  • apps/fast-rspec/test/oracles/rspec-mocks/skip.txt (create) — Known-failing ids: documented header, zero entries today.
  • apps/fast-rspec/test/oracles/webmock/.sha (create) — Pin for webmock oracle: line 1 = 4273387030f7a3f498f9e4ac5f90f5dde8f68086 (webmock/webmock master head — the repo carries NO release tags in this environment, rubygems latest is 3.34.0; note that in a '# ' comment so the movable pin is visible), no tag line.
  • apps/fast-rspec/test/oracles/webmock/Gemfile (create) — Overlay bundling: fast_rspec path ../.., gem 'webmock', path: ENV.fetch('ORACLE_SRC'), plus gem 'rake' (suite is rake-based).
  • apps/fast-rspec/test/oracles/webmock/run (create) — Suite script: cd "$ORACLE_SRC" && bundle exec rake (webmock's default minitest task). Emits ORACLE RESULT line on completion.
  • apps/fast-rspec/test/oracles/webmock/skip.txt (create) — Known-failing ids: documented header, zero entries today.
  • +20 more files

Verification

./bin/check passed (exit 0) in 47s.

What the editors ran to see the change working, and what they saw:

  • cd apps/fast-rspec && for f in test/oracles/*/run; do bash -n "$f" || exit 1; done && bundle exec rubocop test/oracles/*/Gemfile (pins): exit 0; rubocop: "10 files inspected, no offenses detected" — every run script parses, every overlay Gemfile lint-clean (frozen literals, ordered gems, TargetRubyVersion 3.2)
  • chmod 0755 apps/fast-rspec/test/oracles/*/run && ls -l apps/fast-rspec/test/oracles/*/run (pins): all ten run scripts listed as -rwxr-xr-x — required because the runner's checked_pin aborts unless run is File.executable? and spawns it directly
  • code_grep '^[0-9a-f]{40}$' and '^tag .+' over test/oracles/*.sha (pins): 10 pins on line 1 exactly matching the brief's table (minitest 819e9b5..., rspec-core 3bdf310..., rspec-expectations 555979f..., rspec-mocks 0d6a023..., webmock 4273387..., vcr 5dcacdf..., faker e48d35f..., database_cleaner 23fb7a7..., simplecov 0e35b25..., factory_bot 5d3995...); 9 tag fallback lines, webmock correctly tagless
  • code_web_fetch api.github.com/repos/minitest/minitest/commits/v5.27.0 (pins): the v5.27.0 tag resolves to exactly 819e9b5f873d7ab6d4edfeaed48542279367ff93 ("Branching minitest to version 5.27.0"), so the tag fallback ref in minitest/.sha is real
  • cd apps/fast-rspec && bundle exec rake test (runner): 44 tests, 0 failures
  • cd apps/fast-rspec && bundle exec rake -T test:oracles (runner): rake test:oracles # Run upstream suites (rspec-core, minitest, factory_bot, …) against FastRspec (ORACLE= for one)
  • cd apps/fast-rspec && bundle exec rubocop test/oracles/support test/unit/oracles_runner_test.rb Rakefile (runner): 4 files inspected, 13 offenses — 5 in test/oracles/support/runner.rb (Metrics/AbcSize 17.35/17 + 4 correctable), 8 in test/unit/oracles_runner_test.rb (2 Met

…(truncated)

Part 1/2 of a goal

  • developerz-ai/fast_rspec (no PR yet)

🤖 Opened by developerz.ai, an automated agent, not a human, on developerz-ai/fast_rspec.

Raised automatically by a developerz.ai maintainer lane.

🤖 developerz.ai, an automated agent, not a human — this PR is one slice of a stacked plan.
Dispatched automatically by a developerz.ai maintainer lane
Raised automatically by the developerz.ai raw task triage maintainer lane, from manual.
Plan: https://app.developerz.ai/fleet/plans/rtk_bfca831b61444e8cb86731f9d4a8390f
Part 1/2
BYOK spend: unknown (unmetered passes).

Replace the abort at apps/fast-rspec/Rakefile:17-18 with a real oracle
runner: it discovers test/oracles/<gem>/ dirs, fetches the pinned
upstream commit (tag fallback) into tmp/oracles, bundle-installs the
overlay Gemfile, executes each run script with lib/compat unshifted onto
$LOAD_PATH, and classifies every suite as pass / skip(reason) / fail.
Skips (unfetched checkout, install failure, crash before results,
timeout) are reported with their reason and never gate; an un-skipped
failure gates with exit 1. Rows print per suite (pin, counts, %) and
land in tmp/oracles/results.json for docs/compat.md recording.

10 pinned oracles added: minitest 5.27.0, rspec-core v3.13.2,
rspec-expectations v3.13.3, rspec-mocks v3.13.2, webmock (master head,
no release tags), vcr v6.4.0, faker v3.8.0, database_cleaner v2.1.0,
simplecov v0.22.0, factory_bot v6.6.0. Frontends are unimplemented
today, so suites classify skip(did-not-run) — the honest leg-1 output
the compat table consumes.

Offline bootstrap unit tests cover discovery, ORACLE= filter, the
classification matrix, timeouts, skip.txt and exit codes via fake
local-pin oracle dirs; no network or git in the unit suite. AGENTS.md
commands table updated to the implemented behavior.

Dz-Task-Id: tsk_b5b49c53d57ce7796cd234dadce53ed9
@developerz-ai

developerz-ai Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Implement the oracle suite execution path

Work landed on #31 — watching it toward merge.

3h46m · 474 tool calls · 134 messages


🤖 developerz.ai, an automated agent, not a human — developerz-ai[bot]

@developerz-ai developerz-ai Bot added the dz/fleet-pr Opened by developerz.ai, an automated agent, from a fleet run. label Nov 12, 2026
@developerz-ai

developerz-ai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

🟡 Reviewed

9 actionable comment(s) · 2 refuted by verification · 1 suppressed (anti-noise) · grounded on your code

⚠️ 1 config note(s): some test files were not shown to the reviewer — they were read at the reviewed commit, but did not fit the tests section; the least-changed ones were left out

🤖 developerz.ai — automated review, running on your model and your box.

@developerz-ai

developerz-ai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Waiting for a review of this head. Nothing is needed from you right now.

  • Red on this head: no check is failing
  • What was done: CI has finished. No review verdict covers this head yet, or one still holds it.
  • What happens next: The platform asks for its own review when none is coming; a human reviewer who requested changes is waited for. It is handed to a person if nothing arrives.

Status for head c6049e0ba08b. This comment is updated in place.

🤖 Posted by developerz.ai — an automated agent, not a human.

@developerz-ai developerz-ai Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary — 45 file(s), 15 finding(s).

Critical 0 · Major 4 · Minor 11 · Nit 0

Adds pinned upstream RSpec oracle suites (rspec-core/expectations/mocks: .sha, overlay Gemfile, run script, skip.txt). The overlay Gemfiles point the fast_rspec path gem one directory short, so every install fails and every suite lands on skip — no parity coverage yet.

File Findings
apps/fast-rspec/test/oracles/database_cleaner/Gemfile 1 major
apps/fast-rspec/test/oracles/minitest/run 1 minor
apps/fast-rspec/test/oracles/rspec-core/.sha 1 minor
apps/fast-rspec/test/oracles/rspec-core/Gemfile 1 major
apps/fast-rspec/test/oracles/rspec-core/run 1 minor
apps/fast-rspec/test/oracles/support/compat_shadow.rb 1 major
apps/fast-rspec/test/oracles/support/runner.rb 1 major, 2 minor
apps/fast-rspec/test/unit/oracles_runner_test.rb 6 minor
  • some test files were not shown to the reviewer — they were read at the reviewed commit, but did not fit the tests section; the least-changed ones were left out

🤖 developerz.ai review — automated, running on your model and your box. What is this?

@developerz-ai

developerz-ai Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

File fence widened for a findings fix. The fix_pr follow-up for this pull request needed to change file(s) this PR did not touch, to fix what its review thread requires (an existing file the finding names, or a new module it needs) — review thread: PRRT_kwDOVeg6bs62a_9_:

  • apps/fast-rspec/test/oracles/support/totals.rb

Why: Thread PRRT_kwDOVeg6bs62a_9_ asks to extract the suite-totals -> ORACLE RESULT parsing out of the ten bash run scripts into "one shared helper under test/oracles/support" and to assert it in the unit suite. The shared helper is a new file next to support/runner.rb and support/compat_shadow.rb; without it the parsing regexes stay untested in bash only.

The platform granted it under the same area lock a new task takes (no other in-flight task holds these files), after checking the path is not a secret, CI configuration or a human-merge path.

Posted automatically by the developerz bot (AI-authored).

…mfile (+27 more)

Follow-up intent: Address review findings on #31

Dz-Task-Id: tsk_0565342e341cda0412bae391d836b5ad

@developerz-ai developerz-ai Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary — 28 file(s), 9 finding(s).

Critical 0 · Major 0 · Minor 9 · Nit 0

Adds the oracle runner (fetch, bundle install, run, classify), a shared totals parser and the compat shadow, with offline unit coverage. Risk: the rspec oracles now fetch the monorepo while their pins/overlays still target the split repos, and a pass row proves nothing about fast_rspec.

File Findings
apps/fast-rspec/test/oracles/minitest/run 1 minor
apps/fast-rspec/test/oracles/rspec-core/Gemfile 1 minor
apps/fast-rspec/test/oracles/support/compat_shadow.rb 2 minor
apps/fast-rspec/test/oracles/support/runner.rb 3 minor
apps/fast-rspec/test/oracles/support/totals.rb 1 minor
apps/fast-rspec/test/unit/oracles_runner_test.rb 1 minor
9 non-blocking finding(s) below major (no thread on a re-review — only blocking findings open threads)
  • apps/fast-rspec/test/oracles/minitest/run:31 minor · docs The header at line 5 says "run scripts never touch RUBYOPT or the bundle themselves", but this change adds RUBYOPT= here, so the file now contradicts itself about env handling. Update line 5 to state the one exception: RUBYOPT is cleared for the plain-Ruby totals helper (per the note at lines 29-…
  • apps/fast-rspec/test/oracles/rspec-core/Gemfile:5 minor · bug gem 'aruba' is declared only here but the suite runs bundle exec rspec in the rspec-core directory of the upstream monorepo, whose specs shell out through aruba; without it in the bundle the specs fail to load and the oracle silently degrades to a skip. Keep the rspec-expectations and rspec-moc…
  • apps/fast-rspec/test/oracles/support/compat_shadow.rb:11 minor · test No test executes this shim — the offline suite stubs run scripts as /bin/sh — so removing require 'bundler/setup' here would LoadError every bundle exec oracle run into skip(crash) while all unit tests stay green. Pin the order offline: spawn ruby with RUBYOPT pointing at this shim against a tm…
  • apps/fast-rspec/test/oracles/support/compat_shadow.rb:15 minor · test The shim requires fast_rspec but leaves no trace in the suite output, so the runner cannot tell whether the suite actually ran the shadowed code — a shadow that does not take effect still yields a 'pass' row (the unit test's own PASS_SCRIPT, which contains no fast_rspec at all, is already classifie…
  • apps/fast-rspec/test/oracles/support/runner.rb:194 minor · bug totals.rb can legally print a negative pass count (rspec load errors exceeding passing examples, e.g. '3 examples, 0 failures, 5 errors occurred outside of examples' → pass=-2), which this pass=(\d+) regex rejects, so the runner classifies that suite as skip('no ORACLE RESULT after exit 0') inste…
  • apps/fast-rspec/test/oracles/support/runner.rb:264 minor · bug skip.txt is only ever looked up with the suite-level id '<​oracle>:<​stage>', but the skip.txt files this PR adds document per-example known-failing ids, so a documented failure never matches and the suite still ends 'fail' with exit 1. Either look the failing example ids up or make the ids stage-lev…
  • apps/fast-rspec/test/oracles/support/runner.rb:349 minor · bug One suite that cannot be spawned at all (no git, bundle or bash on PATH: Errno::ENOENT from spawn) raises out of the runner: no table, no tmp/oracles/results.json, and the remaining oracles never run. Wrap the per-suite call so such a suite classifies as skip with its reason.
  • apps/fast-rspec/test/oracles/support/totals.rb:107 minor · bug The helper reads the suite log as UTF-8, so a suite that prints a byte that is not valid UTF-8 makes the regexp match raise ArgumentError (invalid byte sequence in UTF-8); the run script then exits without an ORACLE RESULT line and the oracle is classified as skip, silently losing a suite that actu…
  • apps/fast-rspec/test/unit/oracles_runner_test.rb:704 minor · test The fake run scripts print the RESULT line by hand (oracles_runner_test.rb:22), so nothing pins the shell→totals→runner contract this helper creates: a run script that calls totals.rb with the wrong --dialect prints no totals line and the runner classifies that oracle as skip. Add one oracle whose…
  • some test files were not shown to the reviewer — they were read at the reviewed commit, but did not fit the tests section; the least-changed ones were left out

🤖 developerz.ai review — automated, running on your model and your box. What is this?

@developerz-ai
developerz-ai Bot merged commit 8996b13 into main Nov 12, 2026
9 checks passed
@developerz-ai
developerz-ai Bot deleted the dz/implement-the-oracle-suite-execution-path--b5b49c53 branch November 12, 2026 02:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dz/fleet-pr Opened by developerz.ai, an automated agent, from a fleet run.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants