Skip to content

Refuse package mirroring from the CLI while scanning is enabled - #493

Open
abhinavgautam01 wants to merge 1 commit into
git-pkgs:mainfrom
abhinavgautam01:fix/mirror-cli-scanning-478
Open

abhinavgautam01 wants to merge 1 commit into
git-pkgs:mainfrom
abhinavgautam01:fix/mirror-cli-scanning-478

Conversation

@abhinavgautam01

Copy link
Copy Markdown
Contributor

Closes #478

Problem

With scanning.enabled, proxy mirror already refused container images and pointed at the mirror API, but packages from every other ecosystem were still fetched and cached. server.NewMirrorProxy never sets proxy.Scanners, so those artifacts skipped the pre-cache scan (handler.go:594). serve then served them from the cache as ordinary hits, so a proxy configured to scan everything could end up with unscanned artifacts in its cache.

Turning scanning on for the CLI is not an option. Scanners pull the staged bytes from the running server through the signed /_internal/scan-fetch URL and the CLI does not start a server.

Changes

  • internal/mirror: new Mirror.RefusePackages field. When set, every item Images does not take is recorded as failed with that reason and logged as mirror refused, without being fetched. Images still go to Images first, so they keep their own refusal.

  • cmd/proxy: the inline image check becomes refuseUnscannedMirroring, which sets both refusals while scanning is enabled. Both reasons are named constants and the package one carries the same pointer to the mirror API as the image one:

    packages cannot be mirrored by this command while scanning is enabled; use the mirror API of the running server

  • Docs: the README mirror section and the scanning section of docs/configuration.md now explain the restriction. Neither mentioned it before, even for images.

Unchanged:

  • --dry-run still lists what would be mirrored, since it caches nothing.
  • The mirror API is not affected. Its jobs run on the serve proxy and its scanner group.
  • With scanning disabled, the command behaves as before.

Example output with scanning enabled:

Mirror complete: 0 downloaded, 0 skipped (cached), 1 failed, 0 B total

Errors:
npm/lodash@4.17.21: packages cannot be mirrored by this command while scanning is enabled; use the mirror API of the running server

Tests

  • cmd/proxy/mirror_test.go (new): runs the real proxy mirror command in a subprocess with a scanning-enabled config, as the issue asked.
    • A package PURL is refused with the new reason, reported as 0 downloaded and 1 failed and nothing is written to the database or to storage.
    • --dry-run with scanning enabled still lists the package.
    • A unit test checks refuseUnscannedMirroring sets both refusals.
  • internal/mirror/mirror_test.go:
    • Refused packages are never fetched (checked with a fetcher that records every call) and nothing is cached.
    • With both refusals set, a container image still gets the image refusal.

With the refusal disabled, the mirror test fails and lists the npm and crates.io URLs that would have been fetched.

go build, go vet, golangci-lint (v2.13.1), go test -race ./... and Swagger regeneration all pass locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

proxy mirror caches packages unscanned when scanning is enabled

1 participant