Skip to content

Improvement (Bazel Tool): Structural refactoring, query caching, etc - #1863

Open
bd-samratmuk wants to merge 18 commits into
masterfrom
IDETECT-5277
Open

Improvement (Bazel Tool): Structural refactoring, query caching, etc#1863
bd-samratmuk wants to merge 18 commits into
masterfrom
IDETECT-5277

Conversation

@bd-samratmuk

@bd-samratmuk bd-samratmuk commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Description

This PR consolidates several Bazel tool quality improvements and removes duplicated logic. No scanner behaviour changes are intended, except for the two deliberate fixes noted below.

Key Changes

Centralised infrastructure exclusions
Bazel toolchain and build-rule repositories that should never appear in the BOM (for example, bazel_tools, rules_java, rules_jvm_external) now come from a single shared definition. This removes drift across exclusion paths and fixes inconsistent handling of rules_shell.

Shared show_repo execution layer
Common logic for batched bazel mod show_repo execution, output splitting, and retry handling has been consolidated into a shared component used by both the BCR extractor and HTTP pipeline.

Typed parsing for labels and module keys
Bazel labels and module graph keys are now represented by dedicated value types rather than repeated string parsing logic, improving readability and maintainability.

Command result deduplication
Read-only Bazel command results are now cached for the lifetime of a single extraction, eliminating duplicate invocations when multiple components request the same query.

Behavioural Fixes

rules_shell exclusion fix
rules_shell is now consistently excluded across all BOM generation paths. On Bazel 9, it is injected as a toolchain dependency and should never appear as a shipped component.

go_repository URL synthesis fix
Fixed an issue where synthesised GitHub URLs for go_repository entries could be dropped when processing multi-repo show_repo responses containing earlier repositories with explicit URLs.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the Bazel v2 tooling to consolidate duplicated parsing/exclusion logic, centralize show_repo execution mechanics, and introduce per-extraction command result caching, while also addressing the stated rules_shell exclusion consistency and go_repository URL synthesis reliability.

Changes:

  • Centralized “infrastructure” repo/module exclusions and reused them across probing, BCR extraction, and pipelines.
  • Introduced shared show_repo batching + fallback execution and typed value objects (BazelLabel, ModuleKey) to reduce ad-hoc string parsing.
  • Added per-extraction memoization of Bazel command outputs and updated tests/fixtures accordingly (including go_repository synthesis guard fix coverage).

Reviewed changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/test/java/com/blackduck/integration/detect/battery/detector/BazelBattery.java Updates battery fixtures usage to reflect executor-level caching (fewer consumed fixtures).
detectable/src/test/java/com/blackduck/integration/detectable/detectables/bazel/v2/unit/ModuleKeyTest.java Adds unit tests for new ModuleKey parsing behavior.
detectable/src/test/java/com/blackduck/integration/detectable/detectables/bazel/v2/unit/BzlmodBcrExtractorTest.java Adds coverage for show_repo parsing of single url= attribute and dependency resolution.
detectable/src/test/java/com/blackduck/integration/detectable/detectables/bazel/v2/unit/BazelVersionCheckerTest.java Updates mocks to new executor method name/semantics.
detectable/src/test/java/com/blackduck/integration/detectable/detectables/bazel/v2/unit/BazelLabelTest.java Adds unit tests for new BazelLabel structural parsing.
detectable/src/test/java/com/blackduck/integration/detectable/detectables/bazel/v2/unit/BazelCommandExecutorTest.java Updates tests for cached/tolerant execution path and strict failure semantics.
detectable/src/test/java/com/blackduck/integration/detectable/detectables/bazel/functional/bazel/pipeline/step/IntermediateStepParseShowRepoToUrlCandidatesTest.java Adds functional coverage for explicit URL extraction + go_repository synthesis guard fix.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/ModuleKey.java Introduces typed module-key parsing (name@version).
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/HttpFamilyProber.java Uses centralized infra exclusions, label parsing, and tolerant execution API.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BzlmodRepoMappingResolver.java Switches label parsing to BazelLabel and centralizes constants via BazelCommandArguments.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BzlmodGraphJsonParser.java Delegates module key parsing to ModuleKey.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BzlmodBcrExtractor.java Refactors show_repo execution via ShowRepoExecutor, adds query-options plumbing, centralizes infra exclusions.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BazelVersionChecker.java Uses tolerant execution API for --version.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BazelV2Extractor.java Passes user query options into BCR extractor; adds pipeline summary logging.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BazelLabel.java Introduces typed label parsing for repo/apparent/canonical forms and module-extension detection.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BazelInfrastructureModules.java Central “infrastructure module prefixes” source and derived exclusion regex.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/v2/BazelEnvironmentAnalyzer.java Uses tolerant execution API for mode detection.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/query/ModCommandBuilder.java Centralizes @ prefix stripping via BazelCommandArguments.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/query/BazelCommandArguments.java Adds shared constants for repo suffixes, module key separator, label markers.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/pipeline/step/ShowRepoExecutor.java New shared implementation for batched show_repo + per-candidate fallback + block splitting.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/pipeline/step/IntermediateStepParseShowRepoToUrlCandidates.java Fixes go_repository synthesis guard to be per-block, not global.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/pipeline/step/IntermediateStepExecuteShowRepoHeuristic.java Refactors to reuse ShowRepoExecutor and centralized constants.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/pipeline/step/BazelVariableSubstitutor.java Exposes query options getter for consistency; minor formatting issue in substitute.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/pipeline/step/BazelCommandExecutor.java Adds per-extraction raw output caching and renames tolerant execution API.
detectable/src/main/java/com/blackduck/integration/detectable/detectables/bazel/pipeline/Pipelines.java Uses centralized infra exclusion regex (includes rules_shell).
build.gradle Changes project version string (SIGQA/issue-specific snapshot).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +70 to +72
public static String exclusionLookaheadRegex() {
return "^(?!(" + String.join("|", INFRASTRUCTURE_PREFIXES) + ")).*$";
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All current and foreseeable infrastructure prefixes are plain alphanumeric+underscore strings with no regex metacharacters; noted as a future-proofing consideration but not applying the change to avoid adding non-obvious ceremony for a hypothetical case.

Comment on lines +35 to +39
// (issued by both HttpFamilyProber's fast path and BzlmodBcrExtractor), without relying on
// Bazel's server-side analysis cache (which may be evicted by intervening commands).
// Keyed on the exact argument list; only successful (non-throwing) results are cached, and each
// caller still applies its own exit-code interpretation to the shared raw output.
private final Map<List<String>, ExecutableOutput> rawOutputCache = new HashMap<>();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cache is scoped to a single extraction, uses small text entries, and is naturally bounded by the number of distinct Bazel commands in a scan. An LRU would add complexity while risking eviction of entries that may still be reused. Not worth doing.

@bd-samratmuk
bd-samratmuk marked this pull request as ready for review September 1, 2026 10:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants