Improvement (Bazel Tool): Structural refactoring, query caching, etc - #1863
Improvement (Bazel Tool): Structural refactoring, query caching, etc#1863bd-samratmuk wants to merge 18 commits into
Conversation
…po batch+fallback (ShowRepoExecutor)
There was a problem hiding this comment.
Pull request overview
This PR refactors the Bazel v2 tooling to consolidate duplicated parsing/exclusion logic, centralize show_repo execution mechanics, and introduce per-extraction command result caching, while also addressing the stated rules_shell exclusion consistency and go_repository URL synthesis reliability.
Changes:
- Centralized “infrastructure” repo/module exclusions and reused them across probing, BCR extraction, and pipelines.
- Introduced shared
show_repobatching + fallback execution and typed value objects (BazelLabel,ModuleKey) to reduce ad-hoc string parsing. - Added per-extraction memoization of Bazel command outputs and updated tests/fixtures accordingly (including
go_repositorysynthesis guard fix coverage).
Reviewed changes
Copilot reviewed 25 out of 26 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/test/java/com/blackduck/integration/detect/battery/detector/BazelBattery.java | Updates battery fixtures usage to reflect executor-level caching (fewer consumed fixtures). |
| detectable/src/test/java/com/blackduck/integration/detectable/detectables/bazel/v2/unit/ModuleKeyTest.java | Adds unit tests for new ModuleKey parsing behavior. |
| detectable/src/test/java/com/blackduck/integration/detectable/detectables/bazel/v2/unit/BzlmodBcrExtractorTest.java | Adds coverage for show_repo parsing of single url= attribute and dependency resolution. |
| detectable/src/test/java/com/blackduck/integration/detectable/detectables/bazel/v2/unit/BazelVersionCheckerTest.java | Updates mocks to new executor method name/semantics. |
| detectable/src/test/java/com/blackduck/integration/detectable/detectables/bazel/v2/unit/BazelLabelTest.java | Adds unit tests for new BazelLabel structural parsing. |
| detectable/src/test/java/com/blackduck/integration/detectable/detectables/bazel/v2/unit/BazelCommandExecutorTest.java | Updates tests for cached/tolerant execution path and strict failure semantics. |
| detectable/src/test/java/com/blackduck/integration/detectable/detectables/bazel/functional/bazel/pipeline/step/IntermediateStepParseShowRepoToUrlCandidatesTest.java | Adds functional coverage for explicit URL extraction + go_repository synthesis guard fix. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/ModuleKey.java | Introduces typed module-key parsing (name@version). |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/HttpFamilyProber.java | Uses centralized infra exclusions, label parsing, and tolerant execution API. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BzlmodRepoMappingResolver.java | Switches label parsing to BazelLabel and centralizes constants via BazelCommandArguments. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BzlmodGraphJsonParser.java | Delegates module key parsing to ModuleKey. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BzlmodBcrExtractor.java | Refactors show_repo execution via ShowRepoExecutor, adds query-options plumbing, centralizes infra exclusions. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BazelVersionChecker.java | Uses tolerant execution API for --version. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BazelV2Extractor.java | Passes user query options into BCR extractor; adds pipeline summary logging. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BazelLabel.java | Introduces typed label parsing for repo/apparent/canonical forms and module-extension detection. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BazelInfrastructureModules.java | Central “infrastructure module prefixes” source and derived exclusion regex. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BazelEnvironmentAnalyzer.java | Uses tolerant execution API for mode detection. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/query/ModCommandBuilder.java | Centralizes @ prefix stripping via BazelCommandArguments. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/query/BazelCommandArguments.java | Adds shared constants for repo suffixes, module key separator, label markers. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/pipeline/step/ShowRepoExecutor.java | New shared implementation for batched show_repo + per-candidate fallback + block splitting. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/pipeline/step/IntermediateStepParseShowRepoToUrlCandidates.java | Fixes go_repository synthesis guard to be per-block, not global. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/pipeline/step/IntermediateStepExecuteShowRepoHeuristic.java | Refactors to reuse ShowRepoExecutor and centralized constants. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/pipeline/step/BazelVariableSubstitutor.java | Exposes query options getter for consistency; minor formatting issue in substitute. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/pipeline/step/BazelCommandExecutor.java | Adds per-extraction raw output caching and renames tolerant execution API. |
| detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/pipeline/Pipelines.java | Uses centralized infra exclusion regex (includes rules_shell). |
| build.gradle | Changes project version string (SIGQA/issue-specific snapshot). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| public static String exclusionLookaheadRegex() { | ||
| return "^(?!(" + String.join("|", INFRASTRUCTURE_PREFIXES) + ")).*$"; | ||
| } |
There was a problem hiding this comment.
All current and foreseeable infrastructure prefixes are plain alphanumeric+underscore strings with no regex metacharacters; noted as a future-proofing consideration but not applying the change to avoid adding non-obvious ceremony for a hypothetical case.
| // (issued by both HttpFamilyProber's fast path and BzlmodBcrExtractor), without relying on | ||
| // Bazel's server-side analysis cache (which may be evicted by intervening commands). | ||
| // Keyed on the exact argument list; only successful (non-throwing) results are cached, and each | ||
| // caller still applies its own exit-code interpretation to the shared raw output. | ||
| private final Map<List<String>, ExecutableOutput> rawOutputCache = new HashMap<>(); |
There was a problem hiding this comment.
The cache is scoped to a single extraction, uses small text entries, and is naturally bounded by the number of distinct Bazel commands in a scan. An LRU would add complexity while risking eviction of entries that may still be reused. Not worth doing.
Description
This PR consolidates several Bazel tool quality improvements and removes duplicated logic. No scanner behaviour changes are intended, except for the two deliberate fixes noted below.
Key Changes
Centralised infrastructure exclusions
Bazel toolchain and build-rule repositories that should never appear in the BOM (for example, bazel_tools, rules_java, rules_jvm_external) now come from a single shared definition. This removes drift across exclusion paths and fixes inconsistent handling of rules_shell.
Shared show_repo execution layer
Common logic for batched bazel mod show_repo execution, output splitting, and retry handling has been consolidated into a shared component used by both the BCR extractor and HTTP pipeline.
Typed parsing for labels and module keys
Bazel labels and module graph keys are now represented by dedicated value types rather than repeated string parsing logic, improving readability and maintainability.
Command result deduplication
Read-only Bazel command results are now cached for the lifetime of a single extraction, eliminating duplicate invocations when multiple components request the same query.
Behavioural Fixes
rules_shell exclusion fix
rules_shell is now consistently excluded across all BOM generation paths. On Bazel 9, it is injected as a toolchain dependency and should never appear as a shipped component.
go_repository URL synthesis fix
Fixed an issue where synthesised GitHub URLs for go_repository entries could be dropped when processing multi-repo show_repo responses containing earlier repositories with explicit URLs.