[DX-4492] Speed Up ocr2keeper Tests#22942
Open
kalverra wants to merge 6 commits into
Open
Conversation
product-security-plaid-production
Bot
requested review from
ChrisAmora,
erikburt,
se3000 and
timothyF95
Contributor
|
👋 kalverra, thanks for creating this pull request! To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team. Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks! |
Contributor
|
I see you updated files related to
|
Contributor
|
✅ No conflicts with other open PRs targeting |
kalverra
changed the title
ocr2keeper Testsocr2keeper Tests
sebawo
previously approved these changes
Jun 25, 2026
Contributor
|
@kalverra please check failing CI checks. |
|
Contributor
There was a problem hiding this comment.
Pull request overview
Risk Rating: MEDIUM
This PR aims to reduce runtime for core/services/ocr2/plugins/ocr2keeper tests by tightening simulated-chain mining/polling intervals and parallelizing tests, plus some small refactors/lint cleanups in the automation EVM registry code.
Changes:
- Speed up keeper integration tests by reducing simulated backend mining interval and lowering polling / P2P dial & reconcile intervals; add
t.Parallel()in multiple tests. - Refactor v2.0 automation encoder tests to avoid global mutation (enabling safe parallelism) and harden ABI unpacking with a uint32 overflow guard.
- Add/adjust agent skill docs for speeding up tests and clean up various imports/lint issues.
Scrupulous human review recommended for:
- New/expanded
t.Parallel()usage in integration tests (especially anywhere touching shared globals or shared external resources). - Changes to v2.0 automation report encoding/decoding (
encoder.go) and ABI unpacking (abi.go) since these affect core automation data paths. - Logging/error-handling changes inside loops and goroutines (ensure failures are surfaced deterministically, not via later timeouts).
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/test/.agents/skills/fix-flaky-tests/SKILL.md | Updates agent guidance/structure; introduces new “speed up tests” routing and modifies diagnose profiles. |
| tools/test/.agents/skills/fix-flaky-tests/references/speed-up-tests.md | New doc capturing common tactics/learned optimizations for speeding up tests. |
| core/services/ocr2/plugins/ocr2keeper/integration_test.go | Speeds up keeper v2.0 integration tests (mining interval, log poll interval) and replaces gomega Eventually usage with testify. |
| core/services/ocr2/plugins/ocr2keeper/integration_21_test.go | Speeds up keeper v2.1 integration tests by increasing mining frequency and import tidy-ups. |
| core/services/ocr2/plugins/ocr2keeper/evmregistry/v21/integration_test.go | Adds t.Parallel() to multiple integration tests, shortens mining interval, and tweaks log provider/recoverer test logic. |
| core/services/ocr2/plugins/ocr2keeper/evmregistry/v21/autotelemetry21/custom_telemetry.go | Import ordering cleanup. |
| core/services/ocr2/plugins/ocr2keeper/evmregistry/v21/autotelemetry21/custom_telemetry_test.go | Import ordering cleanup; adds t.Parallel(). |
| core/services/ocr2/plugins/ocr2keeper/evmregistry/v20/registry.go | Lint/initialism and minor variable naming improvements; small loop refactor. |
| core/services/ocr2/plugins/ocr2keeper/evmregistry/v20/registry_test.go | Adds t.Parallel() and switches some assertions to require. |
| core/services/ocr2/plugins/ocr2keeper/evmregistry/v20/log_provider.go | Refactors log unmarshal type assertions and renames vars; affects error logging lines near stale report processing. |
| core/services/ocr2/plugins/ocr2keeper/evmregistry/v20/head.go | Import formatting cleanup. |
| core/services/ocr2/plugins/ocr2keeper/evmregistry/v20/encoder.go | Refactors encoder to use instance-level pack/unpack hooks (avoids global mutation, improves testability). |
| core/services/ocr2/plugins/ocr2keeper/evmregistry/v20/encoder_test.go | Makes tests parallel-safe by removing global mutations; adds more t.Parallel() and uses require. |
| core/services/ocr2/plugins/ocr2keeper/evmregistry/v20/abi.go | Renames failure reason constants + adds uint32 overflow guard on block number conversion. |
| core/services/ocr2/plugins/ocr2keeper/evmregistry/v20/abi_test.go | Adds t.Parallel() broadly and updates tests to new failure-reason names + require. |
| core/services/ocr2/plugins/ocr2keeper/config_test.go | Adds t.Parallel() and uses require for unmarshalling assertions. |
Comments suppressed due to low confidence (1)
tools/test/.agents/skills/fix-flaky-tests/SKILL.md:79
- The loop still references
<diagnose-parallel-iterations>and a Smoke profile, but that section/profile no longer exists. This makes the instructions internally inconsistent and harder to follow.
1. If user doesn't have recent results, plan a run with `<diagnose-parallel-iterations>` (default: **Smoke** profile) then execute it. On sandbox errors, follow `<possible_execution_issues>`.
2. If no issues, escalate along `<diagnose-iterations>` (e.g. Smoke → Standard → Deep), increasing `--iterations` and keeping parallelism per `<diagnose-parallel-iterations>`. Ask the user before **Deep** if wall time will be large. If still clean and no fix was needed, end with findings; if a fix was applied, require at least **Standard** before FIXED.
| @@ -0,0 +1,12 @@ | |||
| Optimize slow tests. Exclude non-deterministic flakes/panics. | |||
|
|
|||
| 1. Replace `t.Sleep()` and coarse polling with `require.Eventually` and tight intervals. | |||
| upkeepID := ocr2keepers.UpkeepIdentifier(r.Id.String()) | ||
| checkBlockNumber, err := c.getCheckBlockNumberFromTxHash(ctx, r.TxHash, upkeepID) | ||
| if err != nil { | ||
| c.logger.Error("error while fetching checkBlockNumber from reorged report log: %w", err) |
| upkeepID := ocr2keepers.UpkeepIdentifier(r.Id.String()) | ||
| checkBlockNumber, err := c.getCheckBlockNumberFromTxHash(ctx, r.TxHash, upkeepID) | ||
| if err != nil { | ||
| c.logger.Error("error while fetching checkBlockNumber from insufficient funds report log: %w", err) |
Comment on lines
302
to
306
| // starting the log recoverer should backfill logs | ||
| go func() { | ||
| if startErr := recoverer.Start(ctx); startErr != nil { | ||
| t.Logf("error starting log provider: %s", startErr) | ||
| t.Fail() | ||
| } | ||
| assert.NoError(t, recoverer.Start(ctx), "error starting log provider") | ||
| }() | ||
| defer recoverer.Close() |
Comment on lines
109
to
112
| // using block number at which the upkeep was registered, | ||
| // before we emitted any logs | ||
| UpdateBlock: uint64(n), | ||
| UpdateBlock: 10, | ||
| }) |
Comment on lines
+252
to
253
| t.Parallel() | ||
| ctx := t.Context() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Speeds up
core/services/ocr2/plugins/ocr2keepertests, the 4th slowest test package incore/.Results
How
cltest.Mineat200msLogPollIntervalfaster at100msDeltaDialandDeltaReconcilefaster at100msConfirmed tests pass and aren't flaky after changes with
25iterations.Also includes various linting fixes and
t.Parallel()calls.