diff --git a/.gitignore b/.gitignore index e0081f4..7a996d2 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ .bin/ +.grite/ diff --git a/README.md b/README.md index 17f79c4..c3333ae 100644 --- a/README.md +++ b/README.md @@ -73,6 +73,65 @@ repomap eval --file rules.yaml repomap eval --expr 'kubernetes.kind == "Secret"' ``` +### `deps` + +Generate dependency graphs for Go, Maven, Gradle, npm, and pnpm projects. + +```bash +# Pretty tree output +repomap deps + +# JSON export via stdout +repomap deps --json > out.json + +# Only scan selected managers +repomap deps --manager go,pnpm + +# Include transitive dependencies +repomap deps --depth 0 +``` + +By default, `deps` auto-detects supported manifests and reads local manifest or +lockfile content without running package-manager commands. It prints direct +dependencies by default; use `--depth 0` for the full graph available from the +local files. + +### `cache-warm` + +Prime the local package caches for dependencies this machine has not checked out. + +```bash +# Download a module and its full transitive closure into GOMODCACHE +repomap cache-warm go github.com/flanksource/clicky@v1.21.14 + +# Take the current version, compile every package, and prove it works offline +repomap cache-warm go github.com/flanksource/commons --build --verify + +# Warm several npm packages into the pnpm store +repomap cache-warm pnpm react@18.2.0 react-dom@18.2.0 +``` + +For each `name@version` spec, repomap creates a throwaway single-dependency +project in a temporary directory, drives the real package manager to download the +closure into the machine's shared cache, then deletes the project. Nothing in the +working tree is touched; what persists is the warmed cache (`GOMODCACHE`, the pnpm +store, or the npm cache). Omit the version to take whatever the manager considers +current — the concrete resolved version is reported back, including Go +pseudo-versions. + +`--build` goes further than downloading. For Go it compiles every package in the +module so `GOCACHE` holds build artifacts and not just source. For npm and pnpm it +lets dependency lifecycle scripts run so native addons are compiled; it does not +run the package's own build script. + +`--verify` proves the result rather than assuming it, replaying the work with the +network disabled (`GOPROXY=off`, or an `--offline` install against a frozen +lockfile), so a cache that could not actually build offline fails loudly. + +Supported managers are `go`, `npm`, and `pnpm`. This is aimed at CI images, +sandboxes, and air-gapped builds, where a later build must succeed with no network +access. + ### `version` Print version, commit hash, build date, and Go version. @@ -81,7 +140,7 @@ Print version, commit hash, build date, and Go version. | Flag | Description | Default | |------|-------------|---------| -| `--format` / `-o` | Output format: `pretty`, `json`, `yaml`, `csv`, `table` | `pretty` | +| `--format` / `--json` / `--yaml` / `--csv` | Output format: `pretty`, `json`, `yaml`, `csv`, `markdown`, `html` | `pretty` | ## Configuration diff --git a/cmd/repomap/cache_warm.go b/cmd/repomap/cache_warm.go new file mode 100644 index 0000000..4d83c81 --- /dev/null +++ b/cmd/repomap/cache_warm.go @@ -0,0 +1,97 @@ +package main + +import ( + "context" + "fmt" + "strings" + + "github.com/flanksource/clicky" + "github.com/flanksource/clicky/api" + depgraph "github.com/flanksource/repomap/deps" +) + +type CacheWarmOptions struct { + // No default: tag — a positional field carrying one has its positional values + // silently discarded. Emptiness is validated in runCacheWarm instead. + Args []string `json:"args" args:"true" help:"Package manager (go, npm, pnpm) followed by one or more name@version specs"` + Build bool `json:"build,omitempty" flag:"build" help:"Compile every package after downloading (Go) or run dependency lifecycle and native builds (npm, pnpm)"` + Verify bool `json:"verify,omitempty" flag:"verify" help:"Replay the warm with the network disabled to prove the cache is complete"` +} + +func (opts CacheWarmOptions) GetName() string { return "cache-warm ..." } + +func (opts CacheWarmOptions) Help() api.Text { + return clicky.Text(`Prime the local package caches for dependencies this machine has not checked out. + +For each name@version spec, repomap creates a throwaway single-dependency +project in a temporary directory, drives the real package manager to download +the dependency's full transitive closure into the machine's shared cache, then +deletes the project. Nothing in the working tree is touched; what persists is +the warmed cache (GOMODCACHE, the pnpm store, or the npm cache). + +Omit the version to take whatever the manager considers current. The concrete +resolved version is reported back, including Go pseudo-versions. + +Use --build to go further than downloading. For Go it compiles every package in +the module so GOCACHE holds the build artifacts, not just the source. For npm and +pnpm it lets dependency lifecycle scripts run so native addons are compiled. It +does not run the package's own build script. + +Use --verify to prove the result rather than assume it: the work is replayed with +the network disabled (GOPROXY=off, or an --offline install against a frozen +lockfile), so a cache that could not actually build offline fails loudly. + +This is aimed at CI images, sandboxes, and air-gapped builds, where a later build +must succeed with no network access. + +EXAMPLES: + repomap cache-warm go github.com/flanksource/clicky@v1.21.14 + repomap cache-warm go github.com/flanksource/commons --build --verify + repomap cache-warm pnpm react@18.2.0 react-dom@18.2.0 + repomap cache-warm npm @flanksource/icons@1.0.0 --verify + repomap cache-warm go github.com/flanksource/clicky@v1.21.14 --json`) +} + +func init() { + cmd := clicky.AddNamedCommandWithContext("cache-warm", rootCmd, CacheWarmOptions{}, runCacheWarm) + cmd.Short = "Warm the Go, npm, or pnpm cache for a dependency and optionally build it" +} + +func runCacheWarm(ctx context.Context, opts CacheWarmOptions) (any, error) { + manager, specs, err := parseCacheWarmArgs(opts.Args) + if err != nil { + return nil, err + } + results, err := depgraph.WarmCache(ctx, depgraph.WarmOptions{ + Manager: manager, + Specs: specs, + Build: opts.Build, + Verify: opts.Verify, + }) + // Returned as a slice rather than api.NewTableFrom so --json keeps the full + // WarmResult — per-step commands, durations, and errors, which is what a CI + // debugging session needs. Pretty output still renders as a table via + // WarmResult's Columns/Row. The error is returned alongside the results so a + // partial run still reports which specs succeeded. + return results, err +} + +// parseCacheWarmArgs splits the positional arguments into the manager and its +// specs. Only managers repomap can actually warm are accepted; maven and gradle +// are scan-only, and image/helm are not package caches. +func parseCacheWarmArgs(args []string) (depgraph.Manager, []string, error) { + if len(args) == 0 { + return "", nil, fmt.Errorf("expected a package manager (go, npm, or pnpm) followed by one or more name@version specs") + } + manager := depgraph.Manager(strings.ToLower(strings.TrimSpace(args[0]))) + switch manager { + case depgraph.ManagerGo, depgraph.ManagerNPM, depgraph.ManagerPNPM: + default: + return "", nil, fmt.Errorf("cache warming does not support %q (expected go, npm, or pnpm)", args[0]) + } + specs := splitCommaArgs(args[1:]) + if len(specs) == 0 { + return "", nil, fmt.Errorf("expected at least one name@version spec to warm for %s", manager) + } + return manager, specs, nil +} diff --git a/cmd/repomap/cache_warm_test.go b/cmd/repomap/cache_warm_test.go new file mode 100644 index 0000000..170f03a --- /dev/null +++ b/cmd/repomap/cache_warm_test.go @@ -0,0 +1,114 @@ +package main + +import ( + "context" + "strings" + "testing" + + "github.com/flanksource/clicky" + depgraph "github.com/flanksource/repomap/deps" + "github.com/spf13/cobra" +) + +// runCacheWarmArgs parses argv through a real cobra tree with a stub handler, so +// the assertions cover clicky's struct-tag binding rather than a hand-built +// options struct. +func runCacheWarmArgs(t *testing.T, argv ...string) (CacheWarmOptions, error) { + t.Helper() + var got CacheWarmOptions + root := &cobra.Command{Use: "test"} + clicky.BindAllFlags(root.PersistentFlags(), "tasks", "format") + clicky.AddNamedCommandWithContext("cache-warm", root, CacheWarmOptions{}, func(_ context.Context, opts CacheWarmOptions) (any, error) { + got = opts + return nil, nil + }) + root.SetArgs(argv) + return got, root.Execute() +} + +// The positional args field must carry no default: tag, or clicky silently drops +// the positional values. +func TestCacheWarmBindsPositionalArgsAndFlags(t *testing.T) { + got, err := runCacheWarmArgs(t, "cache-warm", "go", "github.com/acme/lib@v1.2.3", "left-pad@1.3.0", "--build", "--verify") + if err != nil { + t.Fatal(err) + } + want := []string{"go", "github.com/acme/lib@v1.2.3", "left-pad@1.3.0"} + if strings.Join(got.Args, ",") != strings.Join(want, ",") { + t.Fatalf("Args = %v, want %v", got.Args, want) + } + if !got.Build || !got.Verify { + t.Fatalf("Build = %v, Verify = %v, want both true", got.Build, got.Verify) + } +} + +func TestCacheWarmFlagsDefaultOff(t *testing.T) { + got, err := runCacheWarmArgs(t, "cache-warm", "go", "github.com/acme/lib@v1.2.3") + if err != nil { + t.Fatal(err) + } + if got.Build || got.Verify { + t.Fatalf("Build = %v, Verify = %v, want both false", got.Build, got.Verify) + } +} + +func TestParseCacheWarmArgs(t *testing.T) { + cases := []struct { + name string + args []string + wantManager depgraph.Manager + wantSpecs []string + wantErr string + }{ + { + name: "single spec", args: []string{"go", "github.com/acme/lib@v1.2.3"}, + wantManager: depgraph.ManagerGo, wantSpecs: []string{"github.com/acme/lib@v1.2.3"}, + }, + { + name: "several specs", args: []string{"pnpm", "left-pad@1.3.0", "@scope/pkg@2.0.0"}, + wantManager: depgraph.ManagerPNPM, wantSpecs: []string{"left-pad@1.3.0", "@scope/pkg@2.0.0"}, + }, + { + name: "manager casing is normalised", args: []string{"NPM", "left-pad@1.3.0"}, + wantManager: depgraph.ManagerNPM, wantSpecs: []string{"left-pad@1.3.0"}, + }, + {name: "no args", args: nil, wantErr: "manager"}, + {name: "manager but no spec", args: []string{"go"}, wantErr: "spec"}, + // Managers repomap can scan but cannot warm must be rejected by name. + {name: "unwarmable manager", args: []string{"maven", "org.acme:lib@1.0.0"}, wantErr: "maven"}, + {name: "unknown manager", args: []string{"cargo", "serde@1.0.0"}, wantErr: "cargo"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + manager, specs, err := parseCacheWarmArgs(tc.args) + if tc.wantErr != "" { + if err == nil { + t.Fatalf("expected an error mentioning %q", tc.wantErr) + } + if !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("error %q should mention %q", err, tc.wantErr) + } + return + } + if err != nil { + t.Fatal(err) + } + if manager != tc.wantManager { + t.Errorf("manager = %q, want %q", manager, tc.wantManager) + } + if strings.Join(specs, ",") != strings.Join(tc.wantSpecs, ",") { + t.Errorf("specs = %v, want %v", specs, tc.wantSpecs) + } + }) + } +} + +// defaultToScan rewrites anything it does not recognise into `scan ...`, so a +// misregistered name would turn this command into a silent repo scan. +func TestCacheWarmIsNotRewrittenToScan(t *testing.T) { + argv := []string{"cache-warm", "go", "github.com/acme/lib@v1.2.3"} + got := defaultToScan(argv) + if strings.Join(got, " ") != strings.Join(argv, " ") { + t.Fatalf("defaultToScan(%v) = %v, want it unchanged", argv, got) + } +} diff --git a/cmd/repomap/deps.go b/cmd/repomap/deps.go new file mode 100644 index 0000000..6cd524d --- /dev/null +++ b/cmd/repomap/deps.go @@ -0,0 +1,289 @@ +package main + +import ( + "context" + "fmt" + "os" + "strings" + + "github.com/flanksource/clicky" + "github.com/flanksource/clicky/api" + depgraph "github.com/flanksource/repomap/deps" +) + +type DepsOptions struct { + Path string `json:"path" args:"true" help:"Path to scan (defaults to current directory)"` + Manager []string `json:"manager,omitempty" flag:"manager" help:"Dependency manager to include: go, maven, gradle, npm, pnpm, image/docker, helm (repeatable or comma-separated)"` + Depth int `json:"depth,omitempty" flag:"depth" default:"1" help:"Maximum dependency depth (1 = direct only, 0 = unlimited)"` + Filter []string `json:"filter,omitempty" flag:"filter" help:"Dependency filter patterns matched against id, name, version, manager, source, or path; supports comma-separated values and !exclusions"` + Kind []string `json:"kind,omitempty" flag:"kind,k" help:"Filter image/helm dependencies by Kubernetes kind, e.g. HelmRelease,Deployment (MatchItem syntax)"` + Namespace []string `json:"namespace,omitempty" flag:"namespace,n" help:"Filter image/helm dependencies by namespace (MatchItem syntax)"` + Name []string `json:"name,omitempty" flag:"name" help:"Filter image/helm dependencies by resource name (MatchItem syntax)"` + Selector []string `json:"selector,omitempty" flag:"selector,l" help:"Filter image/helm dependencies by label selector, e.g. app=nginx"` + Flat bool `json:"flat,omitempty" flag:"flat" help:"Export a flat node list with edges instead of the dependency tree"` + IncludeIndirect bool `json:"include_indirect,omitempty" flag:"include-indirect" help:"Include Go indirect requirements in --depth 1 listings (ignored at other depths)"` + ShowDuplicates bool `json:"show_duplicates,omitempty" flag:"show-duplicates" help:"Render every occurrence of duplicated dependencies and report duplicates/conflicts (default: collapse to a single resolved node with no duplicate reporting)"` +} + +type DepsUpdateOptions struct { + Args []string `json:"args" args:"true" help:"Optional dependency MatchItem expression followed by optional path"` + Manager []string `json:"manager,omitempty" flag:"manager" help:"Dependency manager to update: go, npm, pnpm, image/docker, helm (repeatable or comma-separated)"` + Filter []string `json:"filter,omitempty" flag:"filter" help:"Dependency filter patterns (MatchItem syntax) matched against name, manager, manager:name, manager:name@version, scope, and version; use path:/file: for manifest paths; supports comma-separated values and !exclusions"` + Kind []string `json:"kind,omitempty" flag:"kind,k" help:"Filter image/helm targets by kind, e.g. HelmRelease,Deployment (MatchItem syntax)"` + Namespace []string `json:"namespace,omitempty" flag:"namespace,n" help:"Filter image/helm targets by namespace (MatchItem syntax)"` + Name []string `json:"name,omitempty" flag:"name" help:"Filter image/helm targets by resource name (MatchItem syntax)"` + Selector []string `json:"selector,omitempty" flag:"selector,l" help:"Filter image/helm targets by label selector, e.g. app=nginx"` + Image []string `json:"image,omitempty" flag:"image" help:"Only container images matching this repo pattern (MatchItem syntax)"` + Chart []string `json:"chart,omitempty" flag:"chart" help:"Only Helm charts matching this name (MatchItem syntax)"` + Latest bool `json:"latest,omitempty" flag:"latest" help:"Resolve each matched dependency to its highest stable version"` + Version string `json:"version,omitempty" flag:"version" help:"Apply this concrete version to all matched dependencies"` + Check bool `json:"check" flag:"check" help:"Resolve and list available updates without prompting or writing"` + DryRun bool `json:"dry_run" flag:"dry-run" help:"Show planned dependency updates without running package-manager commands"` +} + +func (opts DepsOptions) GetName() string { return "deps" } + +func (opts DepsUpdateOptions) GetName() string { return "update [expr] [path]" } + +func (opts DepsOptions) Help() api.Text { + return clicky.Text(`Generate dependency graphs for Go, Maven, Gradle, npm, pnpm, image, and Helm dependencies. + +Repomap auto-detects supported manifests below the selected path and resolves +dependency graphs from local manifest and lockfile content without running +package-manager commands. Image and Helm dependencies are discovered from +git-tracked Kubernetes manifests. + +For Go, Maven, and Gradle, --depth other than 1 resolves transitive dependencies +by shelling out to the package manager (go mod graph, mvn dependency:tree, +gradle dependencies). The tool must be installed; rerun with --depth 1 for +offline direct-only output. + +For Helm charts and container images, --depth other than 1 additionally recurses +into remote dependencies: subcharts are fetched from their Helm repositories and +image base images are resolved from registry labels and Dockerfile FROM +directives (cloning source repos). Fetches are cached under the user cache dir; +failures degrade to warnings. Use --depth 1 to stay fully offline. + +The command uses the normal Clicky output flow. Use --json to write structured +JSON to stdout, for example: + + repomap deps --json > out.json + +By default the JSON export contains the dependency tree under "roots". Use --flat +to export a flat "nodes" list plus "edges" instead of the tree. + +Shared dependencies are collapsed to their resolved (shallowest) occurrence: the +resolved node is tagged with the number of other parents and each parent that +hid a duplicate shows a trailing count. Use --show-duplicates to render every +occurrence instead. + +Image and Helm dependencies discovered from Kubernetes manifests can be narrowed +by resource metadata with --kind/--namespace/--name/--selector (MatchItem +syntax). These filters only match manifest-sourced targets; Chart.yaml chart +directories have no Kubernetes metadata and are excluded whenever any of them is +set. + +EXAMPLES: + repomap deps + repomap deps ./service --manager go + repomap deps --manager npm,pnpm --depth 0 + repomap deps --manager go --depth 0 --flat --json + repomap deps --manager go --include-indirect + repomap deps --depth 0 --manager go --show-duplicates + repomap deps --manager image,helm ./clusters/prod + repomap deps --manager helm -k HelmRelease -n default + repomap deps --filter 'github.com/flanksource/*,!*test*'`) +} + +func (opts DepsUpdateOptions) Help() api.Text { + return clicky.Text(`Update direct package, image, and Helm chart dependencies. + +Dependencies are narrowed with --filter and/or the optional positional expr; +both use commons MatchItem syntax, accept comma-separated values and +!exclusions, and are combined into a single pattern set. Patterns are matched +against dependency names, manager-qualified names, versions, and scopes. +Manifest path matching is explicit with path: or file:. With +no patterns, every matched dependency is considered. Image and Helm targets (from git-tracked +Kubernetes/Flux manifests, including HelmRelease spec.chartRef OCIRepository and +HelmChart sources) can be further narrowed with --kind/--namespace/--name/ +--selector and the --image/--chart name patterns. + +By default repomap prompts for which dependencies and versions to apply. A +dependency declared in several manifests at the same version is confirmed once +and the chosen version written to every occurrence; occurrences sitting at +different current versions are still confirmed separately. Use --latest to +resolve each to its highest stable version, or --version to apply a concrete +version, both non-interactively. Applied updates are staged with git add +(manifests plus lockfiles); --dry-run and --check never stage. + +Use --check to list updateable dependencies without prompting or writing. + +EXAMPLES: + repomap deps update --filter 'github.com/flanksource/*' + repomap deps update --filter '*flanksource*,!*test*' --check + repomap deps update 'github.com/flanksource/*' + repomap deps update '*' --check + repomap deps update --manager helm -k HelmRelease --latest + repomap deps update --manager image -n default --version 1.27.0 + repomap deps update 'path:apps/*/package.json' + repomap deps update 'helm:mission-control' --manager helm + repomap deps update 'npm:@flanksource/*' ./web --manager npm + repomap deps update 'left-pad,!*beta*' --dry-run`) +} + +func init() { + cmd := clicky.AddNamedCommandWithContext("deps", rootCmd, DepsOptions{}, runDeps) + cmd.Short = "Generate dependency graphs for Go, Maven, Gradle, npm, and pnpm projects" + + updateCmd := clicky.AddNamedCommandWithContext("update", cmd, DepsUpdateOptions{}, runDepsUpdate) + updateCmd.Short = "Update direct package, image, and Helm chart dependencies" + + registerDepsDiff(cmd) +} + +func runDeps(ctx context.Context, opts DepsOptions) (*depgraph.Export, error) { + if opts.Path == "" { + opts.Path = "." + } + path, err := resolvePath(opts.Path) + if err != nil { + return nil, err + } + managers, err := parseManagers(opts.Manager) + if err != nil { + return nil, err + } + return depgraph.Scan(ctx, path, depgraph.Options{ + Managers: managers, + MaxDepth: opts.Depth, + Filters: splitCommaArgs(opts.Filter), + Kind: opts.Kind, + Namespace: opts.Namespace, + Name: opts.Name, + Selector: opts.Selector, + Flat: opts.Flat, + IncludeIndirect: opts.IncludeIndirect, + ShowDuplicates: opts.ShowDuplicates, + }) +} + +func runDepsUpdate(ctx context.Context, opts DepsUpdateOptions) (any, error) { + expr, rawPath, err := parseDepsUpdateArgs(opts.Args) + if err != nil { + return nil, err + } + path, err := resolvePath(rawPath) + if err != nil { + return nil, err + } + managers, err := parseUpdateManagers(opts.Manager) + if err != nil { + return nil, err + } + plans, err := depgraph.Update(ctx, path, depgraph.UpdateOptions{ + Managers: managers, + Filters: updateFilters(opts.Filter, expr), + Kind: opts.Kind, + Namespace: opts.Namespace, + Name: opts.Name, + Selector: opts.Selector, + Image: opts.Image, + Chart: opts.Chart, + Latest: opts.Latest, + Version: opts.Version, + Check: opts.Check, + DryRun: opts.DryRun, + }) + if err != nil { + return nil, err + } + return api.NewTableFrom(plans), nil +} + +// updateFilters combines --filter values with the optional positional expr into +// one MatchItem pattern set. Patterns are forwarded raw because depgraph.Update +// already comma-splits and trims them. +func updateFilters(filter []string, expr string) []string { + out := append([]string{}, filter...) + if expr != "" { + out = append(out, expr) + } + return out +} + +// parseDepsUpdateArgs interprets the optional positional [expr] [path]. With one +// argument, an existing directory is treated as the path and anything else as the +// expression, so `deps update ./clusters` and `deps update 'left-pad'` both work. +func parseDepsUpdateArgs(args []string) (expr, path string, err error) { + switch len(args) { + case 0: + return "", ".", nil + case 1: + if isExistingDir(args[0]) { + return "", args[0], nil + } + return args[0], ".", nil + case 2: + return args[0], args[1], nil + default: + return "", "", fmt.Errorf("expected [expr] [path], got %d arguments", len(args)) + } +} + +func isExistingDir(path string) bool { + info, err := os.Stat(path) + return err == nil && info.IsDir() +} + +func parseManagers(values []string) ([]depgraph.Manager, error) { + parts := splitCommaArgs(values) + if len(parts) == 0 { + return nil, nil + } + out := make([]depgraph.Manager, 0, len(parts)) + for _, part := range parts { + manager := depgraph.Manager(strings.ToLower(part)) + switch manager { + case "docker": + out = append(out, depgraph.ManagerImage) + case depgraph.ManagerGo, depgraph.ManagerMaven, depgraph.ManagerGradle, depgraph.ManagerNPM, depgraph.ManagerPNPM, depgraph.ManagerImage, depgraph.ManagerHelm: + out = append(out, manager) + default: + return nil, fmt.Errorf("unsupported dependency manager %q (expected go, maven, gradle, npm, pnpm, image/docker, or helm)", part) + } + } + return out, nil +} + +func parseUpdateManagers(values []string) ([]depgraph.Manager, error) { + parts := splitCommaArgs(values) + if len(parts) == 0 { + return nil, nil + } + out := make([]depgraph.Manager, 0, len(parts)) + for _, part := range parts { + manager := depgraph.Manager(strings.ToLower(part)) + switch manager { + case "docker": + out = append(out, depgraph.ManagerImage) + case depgraph.ManagerGo, depgraph.ManagerNPM, depgraph.ManagerPNPM, depgraph.ManagerImage, depgraph.ManagerHelm: + out = append(out, manager) + default: + return nil, fmt.Errorf("unsupported dependency update manager %q (expected go, npm, pnpm, image/docker, or helm)", part) + } + } + return out, nil +} + +func splitCommaArgs(values []string) []string { + var out []string + for _, value := range values { + for _, part := range strings.Split(value, ",") { + part = strings.TrimSpace(part) + if part != "" { + out = append(out, part) + } + } + } + return out +} diff --git a/cmd/repomap/deps_diff.go b/cmd/repomap/deps_diff.go new file mode 100644 index 0000000..6a29fef --- /dev/null +++ b/cmd/repomap/deps_diff.go @@ -0,0 +1,92 @@ +package main + +import ( + "context" + "fmt" + "strings" + + "github.com/flanksource/clicky" + "github.com/flanksource/clicky/api" + "github.com/spf13/cobra" + depgraph "github.com/flanksource/repomap/deps" +) + +type DepsDiffOptions struct { + Args []string `json:"args" args:"true" required:"true" help:" or .., optionally followed by a path"` + Manager []string `json:"manager,omitempty" flag:"manager" help:"Dependency manager to include: go, maven, gradle, npm, pnpm, image/docker, helm (repeatable or comma-separated)"` + Depth int `json:"depth,omitempty" flag:"depth" default:"1" help:"Maximum dependency depth to compare (1 = direct only, 0 = unlimited)"` + Filter []string `json:"filter,omitempty" flag:"filter" help:"Dependency filter patterns applied to both sides; supports comma-separated values and !exclusions"` +} + +func (opts DepsDiffOptions) GetName() string { return "diff [path]" } + +func (opts DepsDiffOptions) Help() api.Text { + return clicky.Text(`Compare dependency graphs across git revisions. + +Resolves dependencies at a base revision and compares them against the working +tree (default) or a second revision with ref1..ref2. Each non-working-tree side +is checked out into a temporary git worktree so image and Helm discovery work +unchanged. A dirty working tree is compared as-is. + +Pass --depth 0 to compare full transitive graphs (requires the package manager +tools, e.g. go, mvn, gradle); the default --depth 1 compares direct dependencies +offline. + +EXAMPLES: + repomap deps diff HEAD~3 + repomap deps diff main + repomap deps diff v1.0.0..v1.1.0 ./service + repomap deps diff HEAD~5 --manager go --depth 0`) +} + +func registerDepsDiff(parent *cobra.Command) { + diffCmd := clicky.AddNamedCommandWithContext("diff", parent, DepsDiffOptions{}, runDepsDiff) + diffCmd.Short = "Compare dependency graphs across git revisions" +} + +func runDepsDiff(ctx context.Context, opts DepsDiffOptions) (*depgraph.Comparison, error) { + if len(opts.Args) == 0 { + return nil, fmt.Errorf("a git ref or ref1..ref2 range is required") + } + if len(opts.Args) > 2 { + return nil, fmt.Errorf("expected [path], got %d arguments", len(opts.Args)) + } + baseRef, headRef, err := parseRefRange(opts.Args[0]) + if err != nil { + return nil, err + } + path := "." + if len(opts.Args) == 2 { + path = opts.Args[1] + } + path, err = resolvePath(path) + if err != nil { + return nil, err + } + managers, err := parseManagers(opts.Manager) + if err != nil { + return nil, err + } + return depgraph.CompareScan(ctx, path, depgraph.CompareOptions{ + Options: depgraph.Options{ + Managers: managers, + MaxDepth: opts.Depth, + Filters: splitCommaArgs(opts.Filter), + }, + BaseRef: baseRef, + HeadRef: headRef, + }) +} + +// parseRefRange splits a "ref1..ref2" range or returns a single ref with an +// empty head (meaning the working tree). Both sides of a range are required. +func parseRefRange(arg string) (baseRef, headRef string, err error) { + if !strings.Contains(arg, "..") { + return arg, "", nil + } + left, right, _ := strings.Cut(arg, "..") + if left == "" || right == "" { + return "", "", fmt.Errorf("invalid ref range %q: both sides of .. are required", arg) + } + return left, right, nil +} diff --git a/cmd/repomap/deps_diff_test.go b/cmd/repomap/deps_diff_test.go new file mode 100644 index 0000000..b024071 --- /dev/null +++ b/cmd/repomap/deps_diff_test.go @@ -0,0 +1,53 @@ +package main + +import ( + "strings" + "testing" +) + +func TestDepsDiffCommandRegistered(t *testing.T) { + cmd, _, err := rootCmd.Find([]string{"deps", "diff", "HEAD~1"}) + if err != nil { + t.Fatal(err) + } + if cmd == nil || !strings.HasPrefix(cmd.Use, "diff") { + t.Fatalf("expected deps diff command, got %#v", cmd) + } + for _, name := range []string{"depth", "manager", "filter"} { + if flag := cmd.Flags().Lookup(name); flag == nil { + t.Fatalf("%s flag not registered on deps diff", name) + } + } + if depth := cmd.Flags().Lookup("depth"); depth.DefValue != "1" { + t.Fatalf("depth default = %q, want 1", depth.DefValue) + } +} + +func TestParseRefRange(t *testing.T) { + cases := []struct { + arg string + wantBase string + wantHead string + wantErr bool + }{ + {"HEAD", "HEAD", "", false}, + {"v1.0.0..v1.1.0", "v1.0.0", "v1.1.0", false}, + {"..head", "", "", true}, + {"base..", "", "", true}, + } + for _, tc := range cases { + base, head, err := parseRefRange(tc.arg) + if tc.wantErr { + if err == nil { + t.Fatalf("parseRefRange(%q) expected error", tc.arg) + } + continue + } + if err != nil { + t.Fatalf("parseRefRange(%q) unexpected error: %v", tc.arg, err) + } + if base != tc.wantBase || head != tc.wantHead { + t.Fatalf("parseRefRange(%q) = (%q, %q), want (%q, %q)", tc.arg, base, head, tc.wantBase, tc.wantHead) + } + } +} diff --git a/cmd/repomap/deps_test.go b/cmd/repomap/deps_test.go new file mode 100644 index 0000000..647ca00 --- /dev/null +++ b/cmd/repomap/deps_test.go @@ -0,0 +1,254 @@ +package main + +import ( + "context" + "strings" + "testing" + + "github.com/flanksource/clicky" + depgraph "github.com/flanksource/repomap/deps" + "github.com/spf13/cobra" +) + +func TestDepsPositionalPathHonored(t *testing.T) { + var gotPath string + root := &cobra.Command{Use: "test"} + clicky.BindAllFlags(root.PersistentFlags(), "tasks", "format") + clicky.AddNamedCommandWithContext("deps", root, DepsOptions{}, func(_ context.Context, opts DepsOptions) (*depgraph.Export, error) { + gotPath = opts.Path + return &depgraph.Export{}, nil + }) + + root.SetArgs([]string{"deps", "/tmp/some/scan/path", "--json"}) + if err := root.Execute(); err != nil { + t.Fatal(err) + } + if gotPath != "/tmp/some/scan/path" { + t.Fatalf("positional path not bound: opts.Path = %q, want /tmp/some/scan/path", gotPath) + } +} + +func TestParseManagers(t *testing.T) { + got, err := parseManagers([]string{"go,npm", "pnpm", "image", "docker", "helm"}) + if err != nil { + t.Fatal(err) + } + want := []depgraph.Manager{ + depgraph.ManagerGo, + depgraph.ManagerNPM, + depgraph.ManagerPNPM, + depgraph.ManagerImage, + depgraph.ManagerImage, + depgraph.ManagerHelm, + } + if len(got) != len(want) { + t.Fatalf("len = %d, want %d: %#v", len(got), len(want), got) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("manager[%d] = %q, want %q", i, got[i], want[i]) + } + } + if _, err := parseManagers([]string{"ruby"}); err == nil { + t.Fatal("expected unsupported manager error") + } +} + +func TestParseUpdateManagers(t *testing.T) { + got, err := parseUpdateManagers([]string{"go,image", "docker", "helm"}) + if err != nil { + t.Fatal(err) + } + want := []depgraph.Manager{depgraph.ManagerGo, depgraph.ManagerImage, depgraph.ManagerImage, depgraph.ManagerHelm} + if len(got) != len(want) { + t.Fatalf("len = %d, want %d: %#v", len(got), len(want), got) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("manager[%d] = %q, want %q", i, got[i], want[i]) + } + } + if _, err := parseUpdateManagers([]string{"maven"}); err == nil { + t.Fatal("expected unsupported update manager error") + } +} + +func TestDepsNativeResolutionFlagsRemoved(t *testing.T) { + cmd, _, err := rootCmd.Find([]string{"deps"}) + if err != nil { + t.Fatal(err) + } + for _, name := range []string{"mode", "configuration", "strict"} { + if flag := cmd.Flags().Lookup(name); flag != nil { + t.Fatalf("%s flag should be removed from deps listing", name) + } + } +} + +func TestDepsDepthDefault(t *testing.T) { + cmd, _, err := rootCmd.Find([]string{"deps"}) + if err != nil { + t.Fatal(err) + } + flag := cmd.Flags().Lookup("depth") + if flag == nil { + t.Fatal("depth flag not registered") + } + if flag.DefValue != "1" { + t.Fatalf("depth default = %q, want 1", flag.DefValue) + } + if !strings.Contains(flag.Usage, "0 = unlimited") { + t.Fatalf("depth help should document unlimited mode, got %q", flag.Usage) + } +} + +func TestDepsFlatAndIncludeIndirectFlags(t *testing.T) { + cmd, _, err := rootCmd.Find([]string{"deps"}) + if err != nil { + t.Fatal(err) + } + if flag := cmd.Flags().Lookup("flat"); flag == nil { + t.Fatal("flat flag not registered") + } + if flag := cmd.Flags().Lookup("include-indirect"); flag == nil { + t.Fatal("include-indirect flag not registered") + } +} + +func TestDepsUpdateCommandRegistered(t *testing.T) { + cmd, _, err := rootCmd.Find([]string{"deps", "update", "github.com/flanksource/*"}) + if err != nil { + t.Fatal(err) + } + if cmd == nil || !strings.HasPrefix(cmd.Use, "update") { + t.Fatalf("expected deps update command, got %#v", cmd) + } + if flag := cmd.Flags().Lookup("dry-run"); flag == nil { + t.Fatal("dry-run flag not registered") + } + if flag := cmd.Flags().Lookup("check"); flag == nil { + t.Fatal("check flag not registered") + } + manager := cmd.Flags().Lookup("manager") + if manager == nil { + t.Fatal("manager flag not registered") + } + if !strings.Contains(manager.Usage, "go, npm, pnpm, image/docker, helm") { + t.Fatalf("manager help should document update-supported managers, got %q", manager.Usage) + } +} + +// deps update must declare its own --filter, otherwise the flag silently binds +// to clicky's persistent format flag (`--filter string`, a CEL output filter) +// and the MatchItem patterns never reach depgraph.Update. +func TestDepsUpdateFilterFlagShadowsGlobalCELFilter(t *testing.T) { + cmd, _, err := rootCmd.Find([]string{"deps", "update"}) + if err != nil { + t.Fatal(err) + } + flag := cmd.Flags().Lookup("filter") + if flag == nil { + t.Fatal("filter flag not registered on deps update") + } + if got := flag.Value.Type(); got != "stringSlice" { + t.Fatalf("filter flag type = %q, want stringSlice (clicky's global CEL filter is a string)", got) + } + if !strings.Contains(flag.Usage, "MatchItem syntax") { + t.Fatalf("filter help should document MatchItem syntax, got %q", flag.Usage) + } +} + +func TestDepsUpdateFilterFlagBindsToOptions(t *testing.T) { + var got DepsUpdateOptions + root := &cobra.Command{Use: "test"} + clicky.BindAllFlags(root.PersistentFlags(), "tasks", "format") + deps := clicky.AddNamedCommandWithContext("deps", root, DepsOptions{}, func(_ context.Context, _ DepsOptions) (*depgraph.Export, error) { + return &depgraph.Export{}, nil + }) + clicky.AddNamedCommandWithContext("update", deps, DepsUpdateOptions{}, func(_ context.Context, opts DepsUpdateOptions) (any, error) { + got = opts + return nil, nil + }) + + root.SetArgs([]string{"deps", "update", "--filter", "*flanksource*,!*test*", "--filter", "left-pad", "--json"}) + if err := root.Execute(); err != nil { + t.Fatal(err) + } + // cobra's stringSlice comma-splits at parse time; depgraph.Update splits + // again for the positional expr, which arrives as one unsplit string. + want := []string{"*flanksource*", "!*test*", "left-pad"} + if len(got.Filter) != len(want) { + t.Fatalf("Filter = %#v, want %#v", got.Filter, want) + } + for i := range want { + if got.Filter[i] != want[i] { + t.Fatalf("Filter[%d] = %q, want %q", i, got.Filter[i], want[i]) + } + } +} + +func TestUpdateFiltersCombinesFlagAndPositionalExpr(t *testing.T) { + cases := []struct { + name string + filter []string + expr string + want []string + }{ + {name: "neither", want: nil}, + {name: "flag only", filter: []string{"npm:@scope/*"}, want: []string{"npm:@scope/*"}}, + {name: "expr only", expr: "left-pad", want: []string{"left-pad"}}, + { + name: "flag and expr combined", + filter: []string{"*flanksource*", "!*test*"}, + expr: "path:apps/*/package.json", + want: []string{"*flanksource*", "!*test*", "path:apps/*/package.json"}, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := updateFilters(tc.filter, tc.expr) + if len(got) != len(tc.want) { + t.Fatalf("updateFilters(%#v, %q) = %#v, want %#v", tc.filter, tc.expr, got, tc.want) + } + for i := range tc.want { + if got[i] != tc.want[i] { + t.Fatalf("pattern[%d] = %q, want %q", i, got[i], tc.want[i]) + } + } + }) + } +} + +func TestParseDepsUpdateArgs(t *testing.T) { + dir := t.TempDir() + cases := []struct { + name string + args []string + wantExpr string + wantPath string + wantErr bool + }{ + {name: "no args defaults to cwd", wantPath: "."}, + {name: "existing dir is the path", args: []string{dir}, wantPath: dir}, + {name: "non-dir is the expression", args: []string{"left-pad"}, wantExpr: "left-pad", wantPath: "."}, + {name: "expr then path", args: []string{"left-pad", dir}, wantExpr: "left-pad", wantPath: dir}, + {name: "too many args", args: []string{"a", "b", "c"}, wantErr: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + expr, path, err := parseDepsUpdateArgs(tc.args) + if tc.wantErr { + if err == nil { + t.Fatalf("expected error for %#v", tc.args) + } + return + } + if err != nil { + t.Fatal(err) + } + if expr != tc.wantExpr || path != tc.wantPath { + t.Fatalf("parseDepsUpdateArgs(%#v) = (%q, %q), want (%q, %q)", tc.args, expr, path, tc.wantExpr, tc.wantPath) + } + }) + } +} diff --git a/cmd/repomap/images.go b/cmd/repomap/images.go index 40bbf2e..cbf11d1 100644 --- a/cmd/repomap/images.go +++ b/cmd/repomap/images.go @@ -2,18 +2,28 @@ package main import ( "fmt" - "path/filepath" "strings" scannerlog "github.com/argoproj-labs/argocd-image-updater/registry-scanner/pkg/log" - "github.com/flanksource/commons/collections" "github.com/spf13/cobra" "github.com/flanksource/repomap" "github.com/flanksource/repomap/imageupdate" - "github.com/flanksource/repomap/kubernetes" ) +// versionOnly strips an image/chart current value down to its tag/version +// (dropping any registry/repo prefix and digest suffix). +func versionOnly(currentValue string) string { + if i := strings.LastIndex(currentValue, ":"); i >= 0 { + v := currentValue[i+1:] + if at := strings.Index(v, "@"); at >= 0 { + v = v[:at] + } + return v + } + return currentValue +} + // resolveConcurrency bounds how many registry/Helm version lookups run at once. const resolveConcurrency = 8 @@ -34,7 +44,7 @@ func init() { // `images update`. The first four mirror `scan`'s resource filters; --image and // --chart further narrow by image repo / chart name. type imageFilterOptions struct { - Path string `json:"path" args:"true" help:"Path to scan" default:"."` + Path string `json:"path" args:"true" help:"Path to scan (defaults to current directory)"` Kind []string `json:"kind" flag:"kind,k" help:"Filter by kind, e.g. HelmRelease,Deployment (MatchItem syntax)"` Namespace []string `json:"namespace" flag:"namespace,n" help:"Filter by namespace (MatchItem syntax)"` Name []string `json:"name" flag:"name" help:"Filter by resource name (MatchItem syntax)"` @@ -44,8 +54,9 @@ type imageFilterOptions struct { } // discoverAndFilter resolves the scan path, discovers every image/chart target -// in the repo, and applies the shared resource + image/chart filters. It returns -// the matching targets and the HelmRepository source index for chart resolution. +// in the repo (via the shared imageupdate discovery, which also resolves chart +// sources), and applies the shared resource + image/chart filters. It returns the +// matching targets and the source index for chart resolution. func discoverAndFilter(opts imageFilterOptions) ([]imageupdate.UpdateTarget, *imageupdate.SourceIndex, *repomap.ArchConf, error) { path, err := resolvePath(opts.Path) if err != nil { @@ -56,82 +67,12 @@ func discoverAndFilter(opts imageFilterOptions) ([]imageupdate.UpdateTarget, *im return nil, nil, nil, fmt.Errorf("failed to load config: %w", err) } - targets, sourceIndex, err := discoverTargets(conf, path) + res, err := imageupdate.DiscoverRepoTargets(conf, path) if err != nil { return nil, nil, nil, err } matcher := repomap.NewResourceMatcher(opts.Kind, opts.Namespace, opts.Name, opts.Selector) - targets = filterTargets(targets, matcher, opts) - return targets, sourceIndex, conf, nil -} - -// discoverTargets reads every tracked YAML file, builds the kustomize/Flux tree -// so HelmRepository sources can be resolved through Kustomization namespace -// transformers, indexes the HelmRepositories, and extracts update targets from -// files under the scan prefix. -func discoverTargets(conf *repomap.ArchConf, scanPath string) ([]imageupdate.UpdateTarget, *imageupdate.SourceIndex, error) { - files, err := gitListFiles(conf.RepoPath()) - if err != nil { - return nil, nil, fmt.Errorf("failed to list files: %w", err) - } - - var prefix string - if rel, err := filepath.Rel(conf.RepoPath(), scanPath); err == nil && rel != "." { - prefix = rel + string(filepath.Separator) - } - - // Pass 0: read all tracked YAML (paths are repo-relative POSIX from git). - contents := map[string]string{} - for _, f := range files { - if !kubernetes.IsYaml(f) { - continue - } - content, err := conf.ReadFileWithFallback(f, "") - if err != nil { - continue - } - contents[f] = content - } - - // Pass 1: build the kustomize/Flux tree and the HelmRepository index. - tree := imageupdate.BuildKustomizeTree(contents) - sourceIndex := imageupdate.NewSourceIndex(tree) - - // Pass 2: index sources repo-wide; extract targets only under the scan prefix. - var targets []imageupdate.UpdateTarget - for f, content := range contents { - _ = sourceIndex.IndexHelmRepositories(f, content) - - if prefix != "" && !strings.HasPrefix(f, prefix) { - continue - } - fileTargets, err := imageupdate.ExtractTargets(f, content) - if err != nil { - continue - } - targets = append(targets, fileTargets...) - } - return targets, sourceIndex, nil -} - -func filterTargets(targets []imageupdate.UpdateTarget, matcher repomap.ResourceMatcher, opts imageFilterOptions) []imageupdate.UpdateTarget { - var out []imageupdate.UpdateTarget - for _, t := range targets { - if !matcher.MatchesRef(t.Ref) { - continue - } - if t.Kind == imageupdate.TargetImage && len(opts.Image) > 0 { - if matched, _ := collections.MatchItem(t.Image.GetFullNameWithoutTag(), opts.Image...); !matched { - continue - } - } - if t.Kind == imageupdate.TargetChart && len(opts.Chart) > 0 { - if matched, _ := collections.MatchItem(t.ChartName, opts.Chart...); !matched { - continue - } - } - out = append(out, t) - } - return out + targets := imageupdate.Filter(res.Targets, matcher, opts.Image, opts.Chart) + return targets, res.Index, conf, nil } diff --git a/cmd/repomap/images_list.go b/cmd/repomap/images_list.go index 73df896..4b81a6a 100644 --- a/cmd/repomap/images_list.go +++ b/cmd/repomap/images_list.go @@ -4,6 +4,7 @@ import ( "context" "fmt" + "github.com/Masterminds/semver/v3" "github.com/flanksource/clicky" "github.com/flanksource/clicky/api" "github.com/flanksource/clicky/task" @@ -15,7 +16,7 @@ import ( type ListImagesOptions struct { imageFilterOptions - Check bool `json:"check" flag:"check" help:"Resolve the latest available version from the registry/Helm repo"` + Check bool `json:"check" flag:"check" help:"Resolve the latest stable and pre-release versions from the registry/Helm repo"` } func (opts ListImagesOptions) GetName() string { return "list" } @@ -25,13 +26,13 @@ func (opts ListImagesOptions) Help() api.Text { Discovers apps/v1 workload images and Flux HelmRelease chart versions in git-tracked YAML and lists each with its current version, file, and resource. -Offline by default. With --check, the latest available version is resolved from -the container registry / Helm repository and an update-available column is shown. +Offline by default. With --check, the latest stable and pre-release versions are +resolved from the container registry / Helm repository using semantic versioning. EXAMPLES: repomap images list # all images and charts (offline) repomap images list -k Deployment # only Deployment images - repomap images list -k HelmRelease --check # show latest available chart versions`) + repomap images list -k HelmRelease --check # show latest stable and pre-release chart versions`) } func init() { @@ -39,17 +40,19 @@ func init() { cmd.Short = "List image tags and Helm chart versions in tracked manifests" } -// ImageInfo is one discovered image/chart row. When Checked, Latest and -// UpdateAvailable are populated from the registry/Helm repo. +// ImageInfo is one discovered image/chart row. When Checked, Latest, +// LatestPrerelease, and their update flags are populated from the registry/Helm repo. type ImageInfo struct { - Ref kubernetes.KubernetesRef `json:"ref"` - Kind imageupdate.TargetKind `json:"kind"` - File string `json:"file"` - Current string `json:"current"` - Latest string `json:"latest,omitempty"` - UpdateAvailable bool `json:"update_available"` - Checked bool `json:"-"` - Error string `json:"error,omitempty"` + Ref kubernetes.KubernetesRef `json:"ref"` + Kind imageupdate.TargetKind `json:"kind"` + File string `json:"file"` + Current string `json:"current"` + Latest string `json:"latest,omitempty"` + LatestPrerelease string `json:"latest_prerelease,omitempty"` + UpdateAvailable bool `json:"update_available"` + PrereleaseUpdateAvailable bool `json:"prerelease_update_available"` + Checked bool `json:"-"` + Error string `json:"error,omitempty"` } func (i ImageInfo) Pretty() api.Text { @@ -69,8 +72,10 @@ func (i ImageInfo) Columns() []api.ColumnDef { } if i.Checked { cols = append(cols, - api.Column("latest").Label("Latest").Build(), - api.Column("update").Label("Update").Build(), + api.Column("latest").Label("Latest Stable").Build(), + api.Column("stable_update").Label("Stable Update").Build(), + api.Column("latest_prerelease").Label("Latest Pre-release").Build(), + api.Column("prerelease_update").Label("Pre-release Update").Build(), ) } return cols @@ -84,23 +89,23 @@ func (i ImageInfo) Row() map[string]any { "current": clicky.Text(i.Current, "font-mono"), } if i.Checked { - switch { - case i.Error != "": + if i.Error != "" { row["latest"] = clicky.Text(i.Error, "text-red-600") - row["update"] = clicky.Text("?", "text-muted") - case i.UpdateAvailable: - row["latest"] = clicky.Text(i.Latest, "font-mono text-green-600") - row["update"] = clicky.Text("✓", "text-green-600") - default: - row["latest"] = clicky.Text(i.Latest, "font-mono text-muted") - row["update"] = clicky.Text("—", "text-muted") + row["stable_update"] = clicky.Text("?", "text-muted") + row["latest_prerelease"] = clicky.Text(i.Error, "text-red-600") + row["prerelease_update"] = clicky.Text("?", "text-muted") + } else { + row["latest"] = versionCell(i.Latest, i.UpdateAvailable) + row["stable_update"] = updateCell(i.UpdateAvailable) + row["latest_prerelease"] = versionCell(i.LatestPrerelease, i.PrereleaseUpdateAvailable) + row["prerelease_update"] = updateCell(i.PrereleaseUpdateAvailable) } } return row } func runListImages(opts ListImagesOptions) (any, error) { - targets, sourceIndex, _, err := discoverAndFilter(opts.imageFilterOptions) + targets, sourceIndex, conf, err := discoverAndFilter(opts.imageFilterOptions) if err != nil { return nil, err } @@ -112,7 +117,7 @@ func runListImages(opts ListImagesOptions) (any, error) { if opts.Check { resolver = imageupdate.NewResolver() } - infos, err := buildImageInfos(context.Background(), resolver, targets, sourceIndex, opts.Check) + infos, err := buildImageInfos(context.Background(), resolver, targets, sourceIndex, opts.Check, displayPathFuncForConf(conf)) if err != nil { return nil, err } @@ -123,11 +128,14 @@ func runListImages(opts ListImagesOptions) (any, error) { // offline mapping. With --check each target's version lookup runs as its own // clicky task (concurrently, with live progress); a per-target resolution error // is recorded on the row rather than aborting the whole listing. -func buildImageInfos(ctx context.Context, resolver *imageupdate.Resolver, targets []imageupdate.UpdateTarget, sourceIndex *imageupdate.SourceIndex, check bool) ([]ImageInfo, error) { +func buildImageInfos(ctx context.Context, resolver *imageupdate.Resolver, targets []imageupdate.UpdateTarget, sourceIndex *imageupdate.SourceIndex, check bool, displayPath displayPathFunc) ([]ImageInfo, error) { + if displayPath == nil { + displayPath = func(path string) string { return path } + } if !check { infos := make([]ImageInfo, len(targets)) for i, t := range targets { - infos[i] = baseInfo(t, false) + infos[i] = baseInfo(t, false, displayPath(t.File)) } return infos, nil } @@ -136,8 +144,9 @@ func buildImageInfos(ctx context.Context, resolver *imageupdate.Resolver, target group := task.StartGroup[int]("Resolving image versions", task.WithConcurrency(resolveConcurrency)) for i, t := range targets { idx, target := i, t - group.Add(taskName(target), func(ctx flanksourceContext.Context, tk *task.Task) (int, error) { - infos[idx] = checkInfo(ctx, resolver, sourceIndex, target, tk) + displayFile := displayPath(target.File) + group.Add(taskName(target, displayFile), func(ctx flanksourceContext.Context, tk *task.Task) (int, error) { + infos[idx] = checkInfo(ctx, resolver, sourceIndex, target, displayFile, tk) return idx, nil }) } @@ -147,14 +156,15 @@ func buildImageInfos(ctx context.Context, resolver *imageupdate.Resolver, target return infos, nil } -func baseInfo(t imageupdate.UpdateTarget, checked bool) ImageInfo { - return ImageInfo{Ref: t.Ref, Kind: t.Kind, File: t.File, Current: t.CurrentValue, Checked: checked} +func baseInfo(t imageupdate.UpdateTarget, checked bool, displayFile string) ImageInfo { + return ImageInfo{Ref: t.Ref, Kind: t.Kind, File: displayFile, Current: t.CurrentValue, Checked: checked} } -// checkInfo resolves a single target's latest version, recording any failure on -// the row. Resolution errors do not fail the task — the listing reports them. -func checkInfo(ctx context.Context, resolver *imageupdate.Resolver, sourceIndex *imageupdate.SourceIndex, t imageupdate.UpdateTarget, tk *task.Task) ImageInfo { - info := baseInfo(t, true) +// checkInfo resolves a single target's latest stable and pre-release versions, +// recording any failure on the row. Resolution errors do not fail the task; the +// listing reports them. +func checkInfo(ctx context.Context, resolver *imageupdate.Resolver, sourceIndex *imageupdate.SourceIndex, t imageupdate.UpdateTarget, displayFile string, tk *task.Task) ImageInfo { + info := baseInfo(t, true, displayFile) if t.Kind == imageupdate.TargetChart { if err := sourceIndex.Resolve(&t); err != nil { tk.Warnf("source unresolved: %v", err) @@ -162,22 +172,69 @@ func checkInfo(ctx context.Context, resolver *imageupdate.Resolver, sourceIndex return info } } - tk.Infof("looking up latest version") - latest, err := resolver.ResolveLatest(ctx, t) + tk.Infof("looking up latest stable and pre-release versions") + latest, err := resolver.ResolveLatestVersions(ctx, t) if err != nil { tk.Errorf("%v", err) info.Error = err.Error() return info } - info.Latest = latest - info.UpdateAvailable = latest != "" && latest != versionOnly(t.CurrentValue) + if latest.Stable == "" && latest.Prerelease == "" { + info.Error = fmt.Sprintf("no semver-matching version found for %s", versionSourceLabel(t)) + tk.Errorf("%s", info.Error) + return info + } + info.Latest = latest.Stable + info.LatestPrerelease = latest.Prerelease + info.UpdateAvailable = semverUpdateAvailable(t.CurrentValue, info.Latest) + info.PrereleaseUpdateAvailable = semverUpdateAvailable(t.CurrentValue, info.LatestPrerelease) return info } +func versionCell(version string, update bool) api.Text { + if version == "" { + return clicky.Text("-", "text-muted") + } + if update { + return clicky.Text(version, "font-mono text-green-600") + } + return clicky.Text(version, "font-mono text-muted") +} + +func updateCell(update bool) api.Text { + if update { + return clicky.Text("✓", "text-green-600") + } + return clicky.Text("—", "text-muted") +} + +func semverUpdateAvailable(currentValue, latest string) bool { + if latest == "" { + return false + } + current := versionOnly(currentValue) + currentSemver, currentErr := semver.NewVersion(current) + latestSemver, latestErr := semver.NewVersion(latest) + if currentErr != nil || latestErr != nil { + return latest != current + } + return latestSemver.GreaterThan(currentSemver) +} + +func versionSourceLabel(t imageupdate.UpdateTarget) string { + if t.Kind == imageupdate.TargetChart && !t.IsOCI { + return fmt.Sprintf("chart %q in %s", t.ChartName, t.RepoURL) + } + if t.Kind == imageupdate.TargetImage && t.Image != nil { + return fmt.Sprintf("image %s", t.Image.GetFullNameWithoutTag()) + } + return fmt.Sprintf("%s %q", t.Kind, t.CurrentValue) +} + // taskName builds a descriptive, unique label for a target's resolution task: // the chart/image, the file it lives in, and the namespace when one is known // (the namespace is often imposed by Flux/kustomize and left empty in the file). -func taskName(t imageupdate.UpdateTarget) string { +func taskName(t imageupdate.UpdateTarget, displayFile string) string { var subject string if t.Kind == imageupdate.TargetChart { subject = "chart " + t.ChartName @@ -187,7 +244,10 @@ func taskName(t imageupdate.UpdateTarget) string { subject += " [" + t.ContainerName + "]" } } - subject += " in " + t.File + if displayFile == "" { + displayFile = t.File + } + subject += " in " + displayFile if ns := targetNamespace(t); ns != "" { subject += " (ns " + ns + ")" } diff --git a/cmd/repomap/images_list_test.go b/cmd/repomap/images_list_test.go index 519c10f..bcf38d5 100644 --- a/cmd/repomap/images_list_test.go +++ b/cmd/repomap/images_list_test.go @@ -14,7 +14,7 @@ func TestBuildImageInfos_OfflineHasNoLatest(t *testing.T) { content, _ := os.ReadFile(filepath.Join(conf.RepoPath(), rel)) targets, _ := imageupdate.ExtractTargets(rel, string(content)) - infos, err := buildImageInfos(context.Background(), nil, targets, imageupdate.NewSourceIndex(nil), false) + infos, err := buildImageInfos(context.Background(), nil, targets, imageupdate.NewSourceIndex(nil), false, nil) if err != nil { t.Fatal(err) } @@ -25,8 +25,8 @@ func TestBuildImageInfos_OfflineHasNoLatest(t *testing.T) { if info.Current != "nginx:1.25.3" { t.Errorf("current = %q, want nginx:1.25.3", info.Current) } - if info.Checked || info.Latest != "" { - t.Errorf("offline row should not be checked or have latest: %+v", info) + if info.Checked || info.Latest != "" || info.LatestPrerelease != "" { + t.Errorf("offline row should not be checked or have latest versions: %+v", info) } // offline Columns omit latest/update if len(info.Columns()) != 4 { @@ -39,8 +39,8 @@ func TestBuildImageInfos_CheckFlagsUpdateAvailable(t *testing.T) { content, _ := os.ReadFile(filepath.Join(conf.RepoPath(), rel)) targets, _ := imageupdate.ExtractTargets(rel, string(content)) - resolver := fakeImageResolver([]string{"1.25.3", "1.27.0"}) - infos, err := buildImageInfos(context.Background(), resolver, targets, imageupdate.NewSourceIndex(nil), true) + resolver := fakeImageResolver([]string{"1.25.3", "1.27.0", "1.28.0-beta.1"}) + infos, err := buildImageInfos(context.Background(), resolver, targets, imageupdate.NewSourceIndex(nil), true, nil) if err != nil { t.Fatal(err) } @@ -51,11 +51,17 @@ func TestBuildImageInfos_CheckFlagsUpdateAvailable(t *testing.T) { if info.Latest != "1.27.0" { t.Errorf("latest = %q, want 1.27.0", info.Latest) } + if info.LatestPrerelease != "1.28.0-beta.1" { + t.Errorf("latest prerelease = %q, want 1.28.0-beta.1", info.LatestPrerelease) + } if !info.UpdateAvailable { t.Error("expected update available (1.25.3 -> 1.27.0)") } - if len(info.Columns()) != 6 { - t.Errorf("checked columns = %d, want 6", len(info.Columns())) + if !info.PrereleaseUpdateAvailable { + t.Error("expected prerelease update available (1.25.3 -> 1.28.0-beta.1)") + } + if len(info.Columns()) != 8 { + t.Errorf("checked columns = %d, want 8", len(info.Columns())) } } @@ -65,8 +71,45 @@ func TestBuildImageInfos_CheckNoUpdateWhenCurrentIsLatest(t *testing.T) { targets, _ := imageupdate.ExtractTargets(rel, string(content)) resolver := fakeImageResolver([]string{"1.25.3", "1.24.0"}) - infos, _ := buildImageInfos(context.Background(), resolver, targets, imageupdate.NewSourceIndex(nil), true) + infos, _ := buildImageInfos(context.Background(), resolver, targets, imageupdate.NewSourceIndex(nil), true, nil) if infos[0].UpdateAvailable { t.Errorf("no update expected when 1.25.3 is already newest; got latest=%q", infos[0].Latest) } } + +func TestBuildImageInfos_CheckDoesNotFlagOlderPrerelease(t *testing.T) { + conf, rel := writeRepo(t) + content, _ := os.ReadFile(filepath.Join(conf.RepoPath(), rel)) + targets, _ := imageupdate.ExtractTargets(rel, string(content)) + + resolver := fakeImageResolver([]string{"1.25.3", "1.24.0-rc.1"}) + infos, err := buildImageInfos(context.Background(), resolver, targets, imageupdate.NewSourceIndex(nil), true, nil) + if err != nil { + t.Fatal(err) + } + info := infos[0] + if info.LatestPrerelease != "1.24.0-rc.1" { + t.Errorf("latest prerelease = %q, want 1.24.0-rc.1", info.LatestPrerelease) + } + if info.PrereleaseUpdateAvailable { + t.Errorf("older prerelease should not be an available update: %+v", info) + } +} + +func TestDisplayPathForRepoFileRelativeToWorkingDir(t *testing.T) { + conf, _ := writeRepo(t) + oldWorkingDir := workingDir + t.Cleanup(func() { workingDir = oldWorkingDir }) + + workingDir = filepath.Join(conf.RepoPath(), "apps") + if err := os.MkdirAll(workingDir, 0o755); err != nil { + t.Fatal(err) + } + + if got := displayPathForRepoFile(conf, "apps/api/deploy.yaml"); got != "api/deploy.yaml" { + t.Errorf("inside cwd = %q, want api/deploy.yaml", got) + } + if got := displayPathForRepoFile(conf, "clusters/prod/deploy.yaml"); got != "../clusters/prod/deploy.yaml" { + t.Errorf("outside cwd = %q, want ../clusters/prod/deploy.yaml", got) + } +} diff --git a/cmd/repomap/images_test_helpers.go b/cmd/repomap/images_test_helpers.go new file mode 100644 index 0000000..700b081 --- /dev/null +++ b/cmd/repomap/images_test_helpers.go @@ -0,0 +1,62 @@ +package main + +import ( + "context" + "os" + "os/exec" + "path/filepath" + "testing" + + "github.com/argoproj-labs/argocd-image-updater/registry-scanner/pkg/image" + "github.com/argoproj-labs/argocd-image-updater/registry-scanner/pkg/registry" + "github.com/argoproj-labs/argocd-image-updater/registry-scanner/pkg/registry/mocks" + "github.com/stretchr/testify/mock" + + "github.com/flanksource/repomap" + "github.com/flanksource/repomap/imageupdate" +) + +const deploymentManifest = `apiVersion: apps/v1 +kind: Deployment +metadata: + name: web + namespace: default +spec: + template: + spec: + containers: + - name: web + image: nginx:1.25.3 # keep me +` + +func fakeImageResolver(tags []string) *imageupdate.Resolver { + return &imageupdate.Resolver{ + NewRegistryClient: func(ctx context.Context, img *image.ContainerImage) (registry.RegistryClient, error) { + m := &mocks.RegistryClient{} + m.On("Tags", mock.Anything).Return(tags, nil) + return m, nil + }, + } +} + +// writeRepo creates a temp git repo with one manifest and a repomap conf rooted +// there. The manifest is committed so git ls-files discovers it. +func writeRepo(t *testing.T) (*repomap.ArchConf, string) { + t.Helper() + dir := t.TempDir() + if out, err := exec.Command("git", "-C", dir, "init").CombinedOutput(); err != nil { + t.Fatalf("git init: %v: %s", err, out) + } + rel := "deploy.yaml" + if err := os.WriteFile(filepath.Join(dir, rel), []byte(deploymentManifest), 0o644); err != nil { + t.Fatal(err) + } + if out, err := exec.Command("git", "-C", dir, "add", rel).CombinedOutput(); err != nil { + t.Fatalf("git add: %v: %s", err, out) + } + conf, err := repomap.GetConf(dir) + if err != nil { + t.Fatal(err) + } + return conf, rel +} diff --git a/cmd/repomap/images_update.go b/cmd/repomap/images_update.go index 7452ddf..a31b432 100644 --- a/cmd/repomap/images_update.go +++ b/cmd/repomap/images_update.go @@ -3,19 +3,15 @@ package main import ( "context" "fmt" - "path/filepath" - "strings" + "os" "github.com/flanksource/clicky" "github.com/flanksource/clicky/api" - "github.com/flanksource/clicky/task" - flanksourceContext "github.com/flanksource/commons/context" - - "github.com/flanksource/repomap" - "github.com/flanksource/repomap/imageupdate" - "github.com/flanksource/repomap/kubernetes" + depgraph "github.com/flanksource/repomap/deps" ) +// UpdateImageOptions are the flags for the deprecated `images update` command, +// which now delegates to `deps update --manager image,helm`. type UpdateImageOptions struct { imageFilterOptions Latest bool `json:"latest" flag:"latest" help:"Resolve each target to the highest stable semver"` @@ -26,275 +22,45 @@ type UpdateImageOptions struct { func (opts UpdateImageOptions) GetName() string { return "update" } func (opts UpdateImageOptions) Help() api.Text { - return clicky.Text(`Update container image tags and Helm chart versions in tracked manifests. - -Discovers apps/v1 workload images and Flux HelmRelease chart versions in -git-tracked YAML, resolves the target version from the container registry or -Helm repository, and edits the manifest in place (preserving comments). + return clicky.Text(`DEPRECATED: use 'repomap deps update --manager image,helm' instead. -With --latest the highest stable version is chosen. With --version a specific -version is applied. With neither, an interactive picker lists available versions. +Update container image tags and Helm chart versions in tracked manifests. This +command now delegates to 'deps update', which additionally stages applied edits +with git and resolves HelmRelease spec.chartRef (OCIRepository/HelmChart) charts. EXAMPLES: - repomap images update -n default -k HelmRelease --latest - repomap images update -k Deployment --image nginx --version 1.27.0 - repomap images update -k HelmRelease --dry-run`) + repomap deps update --manager helm -n default -k HelmRelease --latest + repomap deps update --manager image -k Deployment --image nginx --version 1.27.0`) } func init() { - cmd := clicky.AddCommand(imagesCmd, UpdateImageOptions{}, runUpdateImage) - cmd.Short = "Update image tags and Helm chart versions in tracked manifests" + cmd := clicky.AddNamedCommandWithContext("update", imagesCmd, UpdateImageOptions{}, runUpdateImage) + cmd.Short = "(deprecated) Update image tags and Helm chart versions; use 'deps update'" } -// UpdatePlan is one resolved (and possibly applied) version change. -type UpdatePlan struct { - Ref kubernetes.KubernetesRef `json:"ref"` - Kind imageupdate.TargetKind `json:"kind"` - File string `json:"file"` - Field string `json:"field"` - OldValue string `json:"old_value"` - NewValue string `json:"new_value"` - Written bool `json:"written"` - DryRun bool `json:"dry_run"` - Skipped string `json:"skipped,omitempty"` -} +// runUpdateImage maps the legacy image-update flags onto deps.Update with the +// image and helm managers, so the two commands share one resolution/apply path. +func runUpdateImage(ctx context.Context, opts UpdateImageOptions) (any, error) { + fmt.Fprintln(os.Stderr, "warning: 'images update' is deprecated; use 'repomap deps update --manager image,helm'") -func (p UpdatePlan) Pretty() api.Text { - t := p.Ref.Pretty().Space() - if p.Skipped != "" { - return t.Append("skipped: "+p.Skipped, "text-muted") - } - t = t.Append(kubernetes.VersionChange{OldVersion: p.OldValue, NewVersion: p.NewValue}.Pretty()) - switch { - case p.DryRun: - t = t.Space().Append("(dry-run)", "text-yellow-600") - case p.Written: - t = t.Space().Append("written", "text-green-600") - } - return t -} - -func (UpdatePlan) Columns() []api.ColumnDef { - return []api.ColumnDef{ - api.Column("resource").Label("Resource").Build(), - api.Column("file").Label("File").Build(), - api.Column("change").Label("Change").Build(), - api.Column("status").Label("Status").Build(), - } -} - -func (p UpdatePlan) Row() map[string]any { - row := map[string]any{ - "resource": p.Ref.Pretty(), - "file": clicky.Text(p.File, "font-mono"), - "change": kubernetes.VersionChange{OldVersion: p.OldValue, NewVersion: p.NewValue}.Pretty(), - } - switch { - case p.Skipped != "": - row["status"] = clicky.Text(p.Skipped, "text-muted") - case p.DryRun: - row["status"] = clicky.Text("dry-run", "text-yellow-600") - case p.Written: - row["status"] = clicky.Text("written", "text-green-600") - } - return row -} - -func runUpdateImage(opts UpdateImageOptions) (any, error) { - if opts.Latest && opts.Version != "" { - return nil, fmt.Errorf("--latest and --version are mutually exclusive") - } - - targets, sourceIndex, conf, err := discoverAndFilter(opts.imageFilterOptions) + path, err := resolvePath(opts.Path) if err != nil { return nil, err } - if len(targets) == 0 { - return nil, fmt.Errorf("no matching image or chart targets found") - } - - resolver := imageupdate.NewResolver() - ctx := context.Background() - - // Resolve chart sources up front (cheap, no network) so the concurrent - // version lookups have a repo URL to query. - for i := range targets { - if targets[i].Kind == imageupdate.TargetChart { - if err := sourceIndex.Resolve(&targets[i]); err != nil { - return nil, err - } - } - } - - var plans []UpdatePlan - if opts.Latest || opts.Version != "" { - plans = resolveConcurrently(ctx, resolver, conf, targets, opts) - } else { - // Interactive picker must run serially (it prompts the user per target). - for _, t := range targets { - plans = append(plans, planTarget(ctx, resolver, conf, t, opts, nil)) - } - } - return api.NewTableFrom(plans), nil -} - -// resolveConcurrently runs each target's version lookup as its own clicky task, -// then applies the resulting edits serially (concurrent writes to the same file -// would race; resolution is the slow, parallelisable part). -func resolveConcurrently(ctx context.Context, resolver *imageupdate.Resolver, conf *repomap.ArchConf, targets []imageupdate.UpdateTarget, opts UpdateImageOptions) []UpdatePlan { - type resolved struct { - newValue string - skipped string - err error - } - results := make([]resolved, len(targets)) - group := task.StartGroup[int]("Resolving image versions", task.WithConcurrency(resolveConcurrency)) - for i, t := range targets { - idx, target := i, t - group.Add(taskName(target), func(ctx flanksourceContext.Context, tk *task.Task) (int, error) { - newValue, skipped, err := resolveNewValue(ctx, resolver, target, opts, tk) - results[idx] = resolved{newValue, skipped, err} - return idx, nil - }) - } - _, _ = group.GetResults() - - plans := make([]UpdatePlan, len(targets)) - for i, t := range targets { - plans[i] = applyResolved(conf, t, results[i].newValue, results[i].skipped, results[i].err, opts) - } - return plans -} - -// planTarget resolves the new version for a target and applies the edit. tk may -// be nil when not running inside a task. -func planTarget(ctx context.Context, resolver *imageupdate.Resolver, conf *repomap.ArchConf, t imageupdate.UpdateTarget, opts UpdateImageOptions, tk *task.Task) UpdatePlan { - newValue, skipped, err := resolveNewValue(ctx, resolver, t, opts, tk) - return applyResolved(conf, t, newValue, skipped, err, opts) -} - -// resolveNewValue determines the replacement value for a target without writing. -// It returns a skip reason instead of a value when no update applies. -func resolveNewValue(ctx context.Context, resolver *imageupdate.Resolver, t imageupdate.UpdateTarget, opts UpdateImageOptions, tk *task.Task) (newValue, skipped string, err error) { - logf(tk, "resolving version") - newVersion, err := chooseVersion(ctx, resolver, t, opts) - if err != nil { - return "", "", err - } - if newVersion == "" { - return "", "no version selected", nil - } - newValue = newVersion - if t.Kind == imageupdate.TargetImage { - newValue, err = resolver.NewImageValue(ctx, t, newVersion) - if err != nil { - return "", "", err - } - } - if newValue == t.CurrentValue { - return "", "already up to date", nil - } - return newValue, "", nil -} - -// applyResolved builds the plan and applies the edit (unless dry-run, skipped, -// or errored). -func applyResolved(conf *repomap.ArchConf, t imageupdate.UpdateTarget, newValue, skipped string, resErr error, opts UpdateImageOptions) UpdatePlan { - plan := UpdatePlan{ - Ref: t.Ref, - Kind: t.Kind, - File: t.File, - Field: t.FieldJSONPath, - OldValue: t.CurrentValue, - DryRun: opts.DryRun, - } - if resErr != nil { - plan.Skipped = resErr.Error() - return plan - } - if skipped != "" { - plan.Skipped = skipped - return plan - } - plan.NewValue = newValue - - absFile := filepath.Join(conf.RepoPath(), t.File) - if _, err := imageupdate.ApplyEdit(absFile, t, newValue, opts.DryRun); err != nil { - plan.Skipped = err.Error() - return plan - } - plan.Written = !opts.DryRun - return plan -} - -func logf(tk *task.Task, format string, args ...any) { - if tk != nil { - tk.Infof(format, args...) - } -} - -// chooseVersion returns the target version per the CLI mode: explicit --version, -// resolved --latest, or an interactive pick from available candidates. -func chooseVersion(ctx context.Context, resolver *imageupdate.Resolver, t imageupdate.UpdateTarget, opts UpdateImageOptions) (string, error) { - if opts.Version != "" { - available, err := resolver.Available(ctx, t) - if err != nil { - return "", err - } - if !contains(available, opts.Version) { - return "", fmt.Errorf("%s: version %q is not available (have: %s)", - t.CurrentValue, opts.Version, strings.Join(available, ", ")) - } - return opts.Version, nil - } - if opts.Latest { - return resolver.ResolveLatest(ctx, t) - } - - available, err := resolver.Available(ctx, t) - if err != nil { - return "", err - } - if len(available) == 0 { - return "", fmt.Errorf("no available versions for %s", t.CurrentValue) - } - return pickVersion(t, available), nil -} - -func pickVersion(t imageupdate.UpdateTarget, available []string) string { - title := fmt.Sprintf("Select version for %s/%s (current %s)", t.Ref.Kind, t.Ref.Name, t.CurrentValue) - choice, ok := clicky.PromptSelect(available, clicky.PromptSelectOptions[string]{ - Title: title, - Render: func(v string) api.Textable { - text := clicky.Text(v) - if v == versionOnly(t.CurrentValue) { - text = text.Space().Append("(current)", "text-muted") - } - return text - }, + plans, err := depgraph.Update(ctx, path, depgraph.UpdateOptions{ + Managers: []depgraph.Manager{depgraph.ManagerImage, depgraph.ManagerHelm}, + Kind: opts.Kind, + Namespace: opts.Namespace, + Name: opts.Name, + Selector: opts.Selector, + Image: opts.Image, + Chart: opts.Chart, + Latest: opts.Latest, + Version: opts.Version, + DryRun: opts.DryRun, }) - if !ok { - return "" - } - return choice -} - -func versionOnly(currentValue string) string { - if i := strings.LastIndex(currentValue, ":"); i >= 0 { - v := currentValue[i+1:] - if at := strings.Index(v, "@"); at >= 0 { - v = v[:at] - } - return v - } - return currentValue -} - -func contains(list []string, v string) bool { - for _, item := range list { - if item == v { - return true - } + if err != nil { + return nil, err } - return false + return api.NewTableFrom(plans), nil } diff --git a/cmd/repomap/images_update_test.go b/cmd/repomap/images_update_test.go deleted file mode 100644 index 0041de0..0000000 --- a/cmd/repomap/images_update_test.go +++ /dev/null @@ -1,143 +0,0 @@ -package main - -import ( - "context" - "os" - "os/exec" - "path/filepath" - "strings" - "testing" - - "github.com/argoproj-labs/argocd-image-updater/registry-scanner/pkg/image" - "github.com/argoproj-labs/argocd-image-updater/registry-scanner/pkg/registry" - "github.com/argoproj-labs/argocd-image-updater/registry-scanner/pkg/registry/mocks" - "github.com/stretchr/testify/mock" - - "github.com/flanksource/repomap" - "github.com/flanksource/repomap/imageupdate" -) - -const deploymentManifest = `apiVersion: apps/v1 -kind: Deployment -metadata: - name: web - namespace: default -spec: - template: - spec: - containers: - - name: web - image: nginx:1.25.3 # keep me -` - -func fakeImageResolver(tags []string) *imageupdate.Resolver { - return &imageupdate.Resolver{ - NewRegistryClient: func(ctx context.Context, img *image.ContainerImage) (registry.RegistryClient, error) { - m := &mocks.RegistryClient{} - m.On("Tags", mock.Anything).Return(tags, nil) - return m, nil - }, - } -} - -// writeRepo creates a temp dir with one manifest and a repomap conf rooted there. -func writeRepo(t *testing.T) (*repomap.ArchConf, string) { - t.Helper() - dir := t.TempDir() - if out, err := exec.Command("git", "-C", dir, "init").CombinedOutput(); err != nil { - t.Fatalf("git init: %v: %s", err, out) - } - rel := "deploy.yaml" - if err := os.WriteFile(filepath.Join(dir, rel), []byte(deploymentManifest), 0o644); err != nil { - t.Fatal(err) - } - conf, err := repomap.GetConf(dir) - if err != nil { - t.Fatal(err) - } - return conf, rel -} - -func TestPlanTarget_LatestDryRun(t *testing.T) { - conf, rel := writeRepo(t) - content, _ := os.ReadFile(filepath.Join(conf.RepoPath(), rel)) - targets, err := imageupdate.ExtractTargets(rel, string(content)) - if err != nil || len(targets) != 1 { - t.Fatalf("extract: %v (%d targets)", err, len(targets)) - } - - resolver := fakeImageResolver([]string{"1.25.3", "1.27.0", "1.28.0-rc.1"}) - plan := planTarget(context.Background(), resolver, conf, targets[0], - UpdateImageOptions{Latest: true, DryRun: true}, nil) - if plan.Skipped != "" { - t.Fatalf("unexpected skip: %s", plan.Skipped) - } - if plan.NewValue != "nginx:1.27.0" { - t.Errorf("new value = %q, want nginx:1.27.0", plan.NewValue) - } - if plan.Written { - t.Error("dry-run must not write") - } - // file must be untouched - after, _ := os.ReadFile(filepath.Join(conf.RepoPath(), rel)) - if string(after) != deploymentManifest { - t.Error("dry-run modified the manifest") - } -} - -func TestPlanTarget_VersionWritesAndPreservesComment(t *testing.T) { - conf, rel := writeRepo(t) - content, _ := os.ReadFile(filepath.Join(conf.RepoPath(), rel)) - targets, _ := imageupdate.ExtractTargets(rel, string(content)) - - resolver := fakeImageResolver([]string{"1.25.3", "1.27.0"}) - plan := planTarget(context.Background(), resolver, conf, targets[0], - UpdateImageOptions{Version: "1.27.0"}, nil) - if !plan.Written { - t.Fatalf("expected written, skipped=%q", plan.Skipped) - } - after, _ := os.ReadFile(filepath.Join(conf.RepoPath(), rel)) - got := splitLine(string(after), 11) - want := " image: nginx:1.27.0 # keep me" - if got != want { - t.Errorf("line 11 = %q, want %q", got, want) - } -} - -func TestPlanTarget_RejectsUnavailableVersion(t *testing.T) { - conf, rel := writeRepo(t) - content, _ := os.ReadFile(filepath.Join(conf.RepoPath(), rel)) - targets, _ := imageupdate.ExtractTargets(rel, string(content)) - - resolver := fakeImageResolver([]string{"1.25.3", "1.27.0"}) - plan := planTarget(context.Background(), resolver, conf, targets[0], - UpdateImageOptions{Version: "9.9.9"}, nil) - if plan.Skipped == "" || !strings.Contains(plan.Skipped, "not available") { - t.Fatalf("expected skip for unavailable version, got skipped=%q written=%v", plan.Skipped, plan.Written) - } -} - -func splitLine(content string, n int) string { - lines := splitLines(content) - if n < 1 || n > len(lines) { - return "" - } - return lines[n-1] -} - -func splitLines(s string) []string { - var out []string - cur := "" - for _, r := range s { - if r == '\n' { - out = append(out, cur) - cur = "" - continue - } - cur += string(r) - } - if cur != "" { - out = append(out, cur) - } - return out -} diff --git a/cmd/repomap/main.go b/cmd/repomap/main.go index 4e689ed..0940a18 100644 --- a/cmd/repomap/main.go +++ b/cmd/repomap/main.go @@ -32,7 +32,7 @@ When run without a subcommand, defaults to 'scan'.`, } func init() { - clicky.BindAllFlags(rootCmd.PersistentFlags(), "format") + clicky.BindAllFlags(rootCmd.PersistentFlags(), "tasks", "format") logger.Configure(logger.Flags{LogToStderr: true, Color: true}) rootCmd.PersistentFlags().StringVar(&workingDir, "cwd", "", "Working directory") @@ -49,17 +49,31 @@ func init() { func main() { defer shutdown.RecoverAndShutdown() - // Default to scan when no subcommand is given - if args := os.Args[1:]; len(args) == 0 || args[0] == "" || args[0][0] == '-' { - rootCmd.SetArgs(append([]string{"scan"}, args...)) - } else if args[0] != "help" && args[0] != "completion" { - if cmd, _, _ := rootCmd.Find(args); cmd == rootCmd { - rootCmd.SetArgs(append([]string{"scan"}, args...)) - } - } + rootCmd.SetArgs(defaultToScan(os.Args[1:])) if err := rootCmd.Execute(); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } } + +// defaultToScan rewrites CLI args so that `scan` is the implicit subcommand: a +// bare invocation, or one that leads with scan flags or a bare path, becomes +// `scan ...`. Help and completion are left untouched so `repomap --help` (and +// `-h`) shows the root command with its full subcommand list rather than scan's +// help. +func defaultToScan(args []string) []string { + if len(args) == 0 { + return []string{"scan"} + } + if args[0] == "-h" || args[0] == "--help" || args[0] == "help" || args[0] == "completion" { + return args + } + if args[0] == "" || args[0][0] == '-' { + return append([]string{"scan"}, args...) + } + if cmd, _, _ := rootCmd.Find(args); cmd == rootCmd { + return append([]string{"scan"}, args...) + } + return args +} diff --git a/cmd/repomap/main_test.go b/cmd/repomap/main_test.go new file mode 100644 index 0000000..57a3a08 --- /dev/null +++ b/cmd/repomap/main_test.go @@ -0,0 +1,31 @@ +package main + +import ( + "reflect" + "testing" +) + +func TestDefaultToScan(t *testing.T) { + cases := []struct { + name string + in []string + want []string + }{ + {"bare invocation runs scan", nil, []string{"scan"}}, + {"leading scan flag prepends scan", []string{"-n", "ns"}, []string{"scan", "-n", "ns"}}, + {"bare path prepends scan", []string{"some/path"}, []string{"scan", "some/path"}}, + {"long help reaches root", []string{"--help"}, []string{"--help"}}, + {"short help reaches root", []string{"-h"}, []string{"-h"}}, + {"help command untouched", []string{"help"}, []string{"help"}}, + {"completion untouched", []string{"completion", "zsh"}, []string{"completion", "zsh"}}, + {"deps subcommand untouched", []string{"deps", "."}, []string{"deps", "."}}, + {"deps help untouched", []string{"deps", "-h"}, []string{"deps", "-h"}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := defaultToScan(tc.in); !reflect.DeepEqual(got, tc.want) { + t.Errorf("defaultToScan(%v) = %v, want %v", tc.in, got, tc.want) + } + }) + } +} diff --git a/cmd/repomap/paths.go b/cmd/repomap/paths.go new file mode 100644 index 0000000..d2bdfd0 --- /dev/null +++ b/cmd/repomap/paths.go @@ -0,0 +1,39 @@ +package main + +import ( + "os" + "path/filepath" + + "github.com/flanksource/repomap" +) + +type displayPathFunc func(string) string + +func displayPathForRepoFile(conf *repomap.ArchConf, repoRelPath string) string { + if conf == nil || repoRelPath == "" { + return repoRelPath + } + absPath := filepath.Join(conf.RepoPath(), filepath.FromSlash(repoRelPath)) + cwd, err := displayCwd() + if err != nil { + return filepath.ToSlash(filepath.Clean(repoRelPath)) + } + rel, err := filepath.Rel(cwd, absPath) + if err != nil { + return filepath.ToSlash(filepath.Clean(repoRelPath)) + } + return filepath.ToSlash(rel) +} + +func displayPathFuncForConf(conf *repomap.ArchConf) displayPathFunc { + return func(repoRelPath string) string { + return displayPathForRepoFile(conf, repoRelPath) + } +} + +func displayCwd() (string, error) { + if workingDir != "" { + return filepath.Abs(workingDir) + } + return os.Getwd() +} diff --git a/cmd/repomap/scan.go b/cmd/repomap/scan.go index b499615..0d8d032 100644 --- a/cmd/repomap/scan.go +++ b/cmd/repomap/scan.go @@ -102,6 +102,7 @@ func runScan(opts ScanOptions) (any, error) { if !opts.Verbose { fm.ScopeMatches = nil } + fm.Path = displayPathForRepoFile(conf, relPath) results = append(results, *fm) } diff --git a/deps/cachewarm.go b/deps/cachewarm.go new file mode 100644 index 0000000..9b9b128 --- /dev/null +++ b/deps/cachewarm.go @@ -0,0 +1,330 @@ +package deps + +import ( + "context" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "time" + + "github.com/flanksource/clicky/task" + flanksourceContext "github.com/flanksource/commons/context" + "github.com/flanksource/repomap/deps/manager/gomod" + "github.com/flanksource/repomap/deps/manager/npm" + "github.com/flanksource/repomap/deps/manager/pnpm" + "github.com/flanksource/repomap/deps/manifest" +) + +// cacheWarmConcurrency bounds how many specs warm at once. GOMODCACHE and the +// pnpm store are both lock-protected against concurrent writers, so the limit is +// about not saturating the network rather than correctness. +const cacheWarmConcurrency = 4 + +// warmers is the single place an ecosystem is wired into cache warming. +var warmers = map[Manager]manifest.Warmer{ + ManagerGo: gomod.Warmer{}, + ManagerNPM: npm.Warmer{}, + ManagerPNPM: pnpm.Warmer{}, +} + +type WarmOptions struct { + Manager Manager + Specs []string + // Build compiles every package of the target (Go) or lets dependency + // lifecycle scripts run so native addons are built (npm, pnpm). + Build bool + // Verify replays the work with the network disabled, turning "a download ran" + // into "this cache can build offline". + Verify bool + Runner CommandRunner +} + +type WarmStep struct { + Name string `json:"name"` + Command string `json:"command"` + Duration time.Duration `json:"duration"` + Error string `json:"error,omitempty"` +} + +type WarmResult struct { + Manager Manager `json:"manager"` + // Spec is the requested name@version; Version is what the manager resolved. + Spec string `json:"spec"` + Name string `json:"name"` + Version string `json:"version,omitempty"` + Packages int `json:"packages,omitempty"` + Cache string `json:"cache,omitempty"` + Built bool `json:"built,omitempty"` + Verified bool `json:"verified,omitempty"` + Steps []WarmStep `json:"steps,omitempty"` + Error string `json:"error,omitempty"` + // SummaryError records a failure to read back what was warmed. The cache is + // still warm when this is set, so it does not fail the spec — but it is + // reported rather than swallowed. + SummaryError string `json:"summary_error,omitempty"` +} + +func (r WarmResult) failed() bool { return r.Error != "" } + +// WarmCache downloads each spec's full dependency closure into the machine's +// shared package cache, using a throwaway project per spec so nothing in the +// user's working tree is touched. +func WarmCache(ctx context.Context, opts WarmOptions) ([]WarmResult, error) { + warmer, ok := warmers[opts.Manager] + if !ok { + return nil, fmt.Errorf("cache warming does not support manager %q (expected go, npm, or pnpm)", opts.Manager) + } + if len(opts.Specs) == 0 { + return nil, fmt.Errorf("cache warming needs at least one name@version spec") + } + runner := opts.Runner + if runner == nil { + runner = ExecRunner{} + // Only preflight when we are really going to exec: an injected runner is a + // test double or a recorder, and need not correspond to a binary on PATH. + if _, err := exec.LookPath(warmer.Binary()); err != nil { + return nil, fmt.Errorf("%s not found on PATH, which is required to warm %s caches: %w", warmer.Binary(), opts.Manager, err) + } + } + + results := make([]WarmResult, len(opts.Specs)) + group := task.StartGroup[int]("Warming package caches", task.WithConcurrency(cacheWarmConcurrency)) + for i, spec := range opts.Specs { + idx, spec := i, spec + group.Add(fmt.Sprintf("%s %s", opts.Manager, spec), func(_ flanksourceContext.Context, tk *task.Task) (int, error) { + results[idx] = warmSpec(ctx, warmer, runner, spec, opts, tk) + switch { + case results[idx].failed(): + tk.Errorf("%s", results[idx].Error) + tk.Failed() + case results[idx].SummaryError != "": + tk.Warnf("%s", results[idx].SummaryError) + tk.Warning() + default: + tk.Success() + } + return idx, nil + }) + } + _, _ = group.GetResults() + + // Each failure carries its own command and the tool's stderr. They are folded + // into the returned error rather than left on the results, because the scratch + // directory is already gone and a caller that only prints the error would + // otherwise have nothing to reproduce from. + var failed []string + for _, result := range results { + if result.failed() { + failed = append(failed, fmt.Sprintf("%s: %s", result.Spec, result.Error)) + } + } + if len(failed) > 0 { + return results, fmt.Errorf("failed to warm %d of %d specs:\n %s", len(failed), len(results), strings.Join(failed, "\n ")) + } + return results, nil +} + +func warmSpec(ctx context.Context, warmer manifest.Warmer, runner CommandRunner, spec string, opts WarmOptions, tk *task.Task) WarmResult { + result := WarmResult{Manager: warmer.Manager(), Spec: spec} + name, version, err := parseWarmSpec(spec) + if err != nil { + result.Error = err.Error() + return result + } + result.Name = name + + dir, err := os.MkdirTemp("", "repomap-cache-warm-*") + if err != nil { + result.Error = err.Error() + return result + } + // The scratch project is disposable: the durable result is what landed in the + // module cache or package store. + defer func() { _ = os.RemoveAll(dir) }() + + probe, err := runProbe(ctx, warmer, runner, dir) + if err != nil { + result.Error = err.Error() + return result + } + steps, err := warmer.Steps(manifest.WarmRequest{ + Dir: dir, + Name: name, + Version: version, + Build: opts.Build, + Verify: opts.Verify, + }, probe) + if err != nil { + result.Error = err.Error() + return result + } + + tk.SetProgress(0, len(steps)) + for i, step := range steps { + tk.Infof("%s", step.Name) + started := time.Now() + err := runWarmStep(ctx, runner, dir, step) + record := WarmStep{Name: step.Name, Command: step.Detail(), Duration: time.Since(started)} + if err != nil { + record.Error = err.Error() + result.Steps = append(result.Steps, record) + result.Error = err.Error() + return result + } + result.Steps = append(result.Steps, record) + tk.SetProgress(i+1, len(steps)) + } + + result.Built = opts.Build + result.Verified = opts.Verify + summarizeWarm(ctx, &result, dir, runner) + return result +} + +// runProbe executes a manager's optional version probe inside the scratch dir and +// returns its trimmed stdout, which Steps consumes as an ordinary input. +func runProbe(ctx context.Context, warmer manifest.Warmer, runner CommandRunner, dir string) (string, error) { + cmd := warmer.Probe() + if cmd == nil { + return "", nil + } + cmd.Dir = dir + result, err := runner.Run(ctx, *cmd) + if err != nil { + return "", commandError(*cmd, result, err) + } + return strings.TrimSpace(result.Stdout), nil +} + +func runWarmStep(ctx context.Context, runner CommandRunner, dir string, step manifest.Step) error { + switch step.Kind { + case manifest.StepWrite: + return os.WriteFile(filepath.Join(dir, step.Path), step.Content, 0o600) + case manifest.StepRemove: + return os.RemoveAll(filepath.Join(dir, step.Path)) + case manifest.StepExec: + result, err := runner.Run(ctx, step.Command) + if err != nil { + return commandError(step.Command, result, err) + } + return nil + default: + return fmt.Errorf("unknown warm step kind %q for step %q", step.Kind, step.Name) + } +} + +// commandError names the exact invocation and the tool's own diagnostics. The +// scratch dir is gone by the time a caller sees this, so the message is the only +// reproduction handle there is. +func commandError(cmd manifest.Command, result CommandResult, err error) error { + if stderr := strings.TrimSpace(result.Stderr); stderr != "" { + return fmt.Errorf("%s: %w: %s", cmd.String(), err, stderr) + } + return fmt.Errorf("%s: %w", cmd.String(), err) +} + +// summarizeWarm reports what actually landed by reading the warmed scratch +// project back through repomap's own manifest resolvers, rather than re-parsing +// go.mod or a lockfile here. The manager did the resolving, so this is also where +// the concrete version (including a Go pseudo-version) is discovered. +// +// It calls discoverOffline/resolveManifest rather than Scan deliberately: Scan +// renders its own task group, which would surface read-back warnings about a +// temporary directory the user never asked about. +func summarizeWarm(ctx context.Context, result *WarmResult, dir string, runner CommandRunner) { + result.Cache = warmCachePath(ctx, result.Manager, runner, dir) + + projects, _, err := discoverOffline(dir, []Manager{result.Manager}) + if err != nil { + result.SummaryError = fmt.Sprintf("could not read back the warmed project: %s", err) + return + } + if len(projects) == 0 { + result.SummaryError = fmt.Sprintf("no %s manifest was produced in the scratch project", result.Manager) + return + } + // MaxDepth 1 keeps the read-back offline — it parses the manifest and lockfile + // the warm just wrote instead of shelling out again. + root, _, err := resolveManifest(ctx, projects[0], Options{MaxDepth: 1, IncludeIndirect: true}) + if err != nil { + result.SummaryError = fmt.Sprintf("could not read back the warmed project: %s", err) + return + } + result.Packages, result.Version = walkWarmed(root, result.Name) +} + +// walkWarmed counts every resolved dependency below root and finds the target's +// version. It walks rather than reading root.Children directly because pnpm nests +// dependencies under a synthetic importer node while go.mod lists them flat. +// +// Only nodes carrying a version are counted, which is what distinguishes a real +// package from pnpm's importer (and from the project root itself). +func walkWarmed(root *Node, name string) (count int, version string) { + seen := map[*Node]bool{} + stack := []*Node{root} + for len(stack) > 0 { + node := stack[len(stack)-1] + stack = stack[:len(stack)-1] + if node == nil || seen[node] { + continue + } + seen[node] = true + for _, child := range node.Children { + if child.Version != "" { + count++ + if version == "" && child.Name == name { + version = child.Version + } + } + stack = append(stack, child) + } + } + return count, version +} + +// warmCachePath asks the manager where its cache lives, so the output names the +// directory that grew. A failure here is not worth reporting: it costs only the +// display of a path. +func warmCachePath(ctx context.Context, manager Manager, runner CommandRunner, dir string) string { + var cmd manifest.Command + switch manager { + case ManagerGo: + cmd = manifest.Command{Dir: dir, Name: "go", Args: []string{"env", "GOMODCACHE"}} + case ManagerPNPM: + cmd = manifest.Command{Dir: dir, Name: "pnpm", Args: []string{"store", "path"}} + case ManagerNPM: + cmd = manifest.Command{Dir: dir, Name: "npm", Args: []string{"config", "get", "cache"}} + default: + return "" + } + result, err := runner.Run(ctx, cmd) + if err != nil { + return "" + } + return strings.TrimSpace(result.Stdout) +} + +// parseWarmSpec splits "name@version". The split uses the last @ at a non-zero +// index so a scoped npm name such as @scope/pkg keeps its leading @. An omitted +// version becomes "latest" and the manager decides what that means; the concrete +// version is read back after warming. +func parseWarmSpec(spec string) (name, version string, err error) { + spec = strings.TrimSpace(spec) + if spec == "" { + return "", "", fmt.Errorf("empty dependency spec: expected name@version") + } + at := strings.LastIndex(spec, "@") + if at <= 0 { + if spec == "@" { + return "", "", fmt.Errorf("invalid dependency spec %q: expected name@version", spec) + } + return spec, "latest", nil + } + name, version = spec[:at], spec[at+1:] + if name == "" || version == "" { + return "", "", fmt.Errorf("invalid dependency spec %q: expected name@version", spec) + } + return name, version, nil +} diff --git a/deps/cachewarm_plan.go b/deps/cachewarm_plan.go new file mode 100644 index 0000000..4847bac --- /dev/null +++ b/deps/cachewarm_plan.go @@ -0,0 +1,90 @@ +package deps + +import ( + "fmt" + "strings" + + "github.com/flanksource/clicky" + "github.com/flanksource/clicky/api" +) + +func (r WarmResult) Pretty() api.Text { + t := clicky.Text(fmt.Sprintf("[%s] %s", r.Manager, r.Name), managerStyle(r.Manager)) + if version := r.displayVersion(); version != "" { + t = t.Space().Append(version, "font-mono text-muted") + } + if r.Error != "" { + return t.Space().Append("failed: "+r.Error, "text-red-600") + } + if r.Packages > 0 { + t = t.Space().Append(fmt.Sprintf("%d packages", r.Packages), "text-muted") + } + t = t.Space().Append(strings.Join(r.badges(), " "), "text-green-600") + if r.Cache != "" { + t = t.Space().Append(r.Cache, "font-mono text-muted") + } + if r.SummaryError != "" { + t = t.Space().Append("("+r.SummaryError+")", "text-yellow-600") + } + return t +} + +func (WarmResult) Columns() []api.ColumnDef { + return []api.ColumnDef{ + api.Column("manager").Label("Manager").Build(), + api.Column("dependency").Label("Dependency").Build(), + api.Column("version").Label("Version").Build(), + api.Column("packages").Label("Packages").Build(), + api.Column("status").Label("Status").Build(), + api.Column("cache").Label("Cache").Build(), + } +} + +func (r WarmResult) Row() map[string]any { + row := map[string]any{ + "manager": clicky.Text(string(r.Manager), managerStyle(r.Manager)), + "dependency": clicky.Text(r.Name, "font-bold text-cyan-600"), + "version": clicky.Text(r.displayVersion(), "font-mono"), + "cache": clicky.Text(r.Cache, "font-mono text-muted"), + } + if r.Packages > 0 { + row["packages"] = clicky.Text(fmt.Sprintf("%d", r.Packages), "text-muted") + } else { + row["packages"] = clicky.Text("") + } + if r.Error != "" { + row["status"] = clicky.Text("failed: "+r.Error, "text-red-600") + return row + } + status := clicky.Text(strings.Join(r.badges(), " "), "text-green-600") + if r.SummaryError != "" { + status = status.Append(" ("+r.SummaryError+")", "text-yellow-600") + } + row["status"] = status + return row +} + +// badges names what the warm actually did, so "warmed" is never confused with +// "compiled" or with "proven to work offline". +func (r WarmResult) badges() []string { + badges := []string{"warmed"} + if r.Built { + badges = append(badges, "built") + } + if r.Verified { + badges = append(badges, "verified offline") + } + return badges +} + +// displayVersion prefers the version the manager resolved, falling back to what +// was requested when the read-back could not determine it. +func (r WarmResult) displayVersion() string { + if r.Version != "" { + return r.Version + } + if _, version, err := parseWarmSpec(r.Spec); err == nil { + return version + } + return "" +} diff --git a/deps/cachewarm_test.go b/deps/cachewarm_test.go new file mode 100644 index 0000000..a1c9ca8 --- /dev/null +++ b/deps/cachewarm_test.go @@ -0,0 +1,272 @@ +package deps + +import ( + "context" + "errors" + "strings" + "testing" +) + +func TestParseWarmSpec(t *testing.T) { + cases := []struct { + spec string + wantName string + wantVersion string + wantErr bool + }{ + {spec: "github.com/acme/lib@v1.2.3", wantName: "github.com/acme/lib", wantVersion: "v1.2.3"}, + // No version means "whatever the manager considers current"; the concrete + // version is read back after warming. + {spec: "github.com/acme/lib", wantName: "github.com/acme/lib", wantVersion: "latest"}, + {spec: "left-pad@1.3.0", wantName: "left-pad", wantVersion: "1.3.0"}, + {spec: "left-pad@^1.3.0", wantName: "left-pad", wantVersion: "^1.3.0"}, + {spec: "left-pad@latest", wantName: "left-pad", wantVersion: "latest"}, + // A scoped npm name leads with @, so splitting must use the last @ and + // ignore one at index 0. + {spec: "@scope/pkg@1.0.0", wantName: "@scope/pkg", wantVersion: "1.0.0"}, + {spec: "@scope/pkg", wantName: "@scope/pkg", wantVersion: "latest"}, + // A Go branch or commit reference must survive untouched. + {spec: "github.com/acme/lib@main", wantName: "github.com/acme/lib", wantVersion: "main"}, + {spec: "", wantErr: true}, + {spec: " ", wantErr: true}, + {spec: "left-pad@", wantErr: true}, + {spec: "@", wantErr: true}, + } + for _, tc := range cases { + t.Run(tc.spec, func(t *testing.T) { + name, version, err := parseWarmSpec(tc.spec) + if tc.wantErr { + if err == nil { + t.Fatalf("parseWarmSpec(%q) = (%q, %q), want an error", tc.spec, name, version) + } + return + } + if err != nil { + t.Fatal(err) + } + if name != tc.wantName || version != tc.wantVersion { + t.Fatalf("parseWarmSpec(%q) = (%q, %q), want (%q, %q)", tc.spec, name, version, tc.wantName, tc.wantVersion) + } + }) + } +} + +// pnpm nests dependencies under a synthetic importer node, so counting +// root.Children would report the importer instead of the packages, and the +// version lookup would miss the target entirely. +func TestWalkWarmedSkipsSyntheticNodes(t *testing.T) { + leftPad := NewNode(ManagerPNPM, "left-pad", "1.3.0") + nested := NewNode(ManagerPNPM, "big-int", "1.0.2") + leftPad.Children = []*Node{nested} + importer := NewNode(ManagerPNPM, ".", "") + importer.Source = "importer" + importer.Children = []*Node{leftPad} + root := NewNode(ManagerPNPM, "scratch", "") + root.Children = []*Node{importer} + + count, version := walkWarmed(root, "left-pad") + if count != 2 { + t.Errorf("count = %d, want 2: the importer carries no version and is not a package", count) + } + if version != "1.3.0" { + t.Errorf("version = %q, want 1.3.0: the target sits below the importer", version) + } +} + +func TestWalkWarmedFindsFlatGoRequires(t *testing.T) { + root := NewNode(ManagerGo, "repomap.local/cachewarm", "") + root.Children = []*Node{ + NewNode(ManagerGo, "rsc.io/quote", "v1.5.2"), + NewNode(ManagerGo, "rsc.io/sampler", "v1.3.0"), + } + count, version := walkWarmed(root, "rsc.io/quote") + if count != 2 || version != "v1.5.2" { + t.Fatalf("walkWarmed = (%d, %q), want (2, v1.5.2)", count, version) + } +} + +func TestWarmCacheRejectsUnsupportedManager(t *testing.T) { + for _, manager := range []Manager{ManagerMaven, ManagerGradle, ManagerHelm, ManagerImage, Manager("cargo")} { + _, err := WarmCache(context.Background(), WarmOptions{ + Manager: manager, + Specs: []string{"something@1.0.0"}, + Runner: &updateFakeRunner{}, + }) + if err == nil { + t.Errorf("manager %q: expected an error", manager) + continue + } + if !strings.Contains(err.Error(), "go, npm, or pnpm") { + t.Errorf("manager %q: error should list the supported managers, got %v", manager, err) + } + } +} + +func TestWarmCacheRequiresSpecs(t *testing.T) { + if _, err := WarmCache(context.Background(), WarmOptions{ + Manager: ManagerGo, + Runner: &updateFakeRunner{}, + }); err == nil { + t.Fatal("expected an error when no specs are given") + } +} + +func TestWarmCacheRunsTheGoStepsInOrder(t *testing.T) { + runner := &updateFakeRunner{succeedByDefault: true} + results, err := WarmCache(context.Background(), WarmOptions{ + Manager: ManagerGo, + Specs: []string{"github.com/acme/lib@v1.2.3"}, + Build: true, + Verify: true, + Runner: runner, + }) + if err != nil { + t.Fatal(err) + } + if len(results) != 1 { + t.Fatalf("want 1 result, got %d", len(results)) + } + result := results[0] + if result.Error != "" { + t.Fatalf("unexpected warm error: %s", result.Error) + } + if result.Name != "github.com/acme/lib" || result.Spec != "github.com/acme/lib@v1.2.3" { + t.Errorf("result identity = %q / %q", result.Name, result.Spec) + } + if !result.Built || !result.Verified { + t.Errorf("Built = %v, Verified = %v, want both true", result.Built, result.Verified) + } + steps := []string{ + "go mod init repomap.local/cachewarm", + "go get github.com/acme/lib/...@v1.2.3", + "go mod download all", + "go build github.com/acme/lib/...", + // The verify replay is the same build with GOPROXY=off, which the argv + // alone does not show; deps/manager/gomod pins the env. + "go build github.com/acme/lib/...", + } + // Reporting where the cache lives runs after the steps, not as one of them. + want := append(append([]string{}, steps...), "go env GOMODCACHE") + if got := runner.ran(); strings.Join(got, "\n") != strings.Join(want, "\n") { + t.Fatalf("commands mismatch\n got: %v\nwant: %v", got, want) + } + if len(result.Steps) != len(steps) { + t.Errorf("recorded %d steps, want %d", len(result.Steps), len(steps)) + } +} + +// The scratch directory is deleted before the error surfaces, so the message is +// the only handle the user has for reproducing the failure. +func TestWarmCacheFailureAbortsRemainingStepsAndNamesTheCommand(t *testing.T) { + runner := &updateFakeRunner{ + succeedByDefault: true, + errors: map[string]error{ + "go get github.com/acme/lib/...@v1.2.3": errors.New("exit status 1"), + }, + responses: map[string]CommandResult{ + "go get github.com/acme/lib/...@v1.2.3": {Stderr: "module github.com/acme/lib: not found"}, + }, + } + results, err := WarmCache(context.Background(), WarmOptions{ + Manager: ManagerGo, + Specs: []string{"github.com/acme/lib@v1.2.3"}, + Build: true, + Runner: runner, + }) + if err == nil { + t.Fatal("WarmCache should report an error when a spec fails, so the CLI exits non-zero") + } + if len(results) != 1 { + t.Fatalf("want 1 result, got %d", len(results)) + } + for _, want := range []string{"go get github.com/acme/lib/...@v1.2.3", "not found"} { + if !strings.Contains(results[0].Error, want) { + t.Errorf("result error %q should contain %q", results[0].Error, want) + } + // The returned error is what a caller that only prints err sees, and the + // scratch dir is gone by then, so the detail has to survive into it too. + if !strings.Contains(err.Error(), want) { + t.Errorf("returned error %q should contain %q", err, want) + } + } + if results[0].Built { + t.Error("Built should stay false when the warm failed") + } + // init ran, get failed, and download/build must not have been attempted. + want := []string{"go mod init repomap.local/cachewarm", "go get github.com/acme/lib/...@v1.2.3"} + if got := runner.ran(); strings.Join(got, "\n") != strings.Join(want, "\n") { + t.Fatalf("commands mismatch\n got: %v\nwant: %v", got, want) + } +} + +// pnpm's probe output decides the build flags, so the orchestrator has to run it +// and feed the result into Steps. +func TestWarmCacheFeedsTheProbeIntoSteps(t *testing.T) { + runner := &updateFakeRunner{ + succeedByDefault: true, + responses: map[string]CommandResult{ + "pnpm --version": {Stdout: "10.7.0\n"}, + }, + } + if _, err := WarmCache(context.Background(), WarmOptions{ + Manager: ManagerPNPM, + Specs: []string{"left-pad@1.3.0"}, + Build: true, + Runner: runner, + }); err != nil { + t.Fatal(err) + } + want := []string{ + "pnpm --version", + "pnpm install --config.dangerouslyAllowAllBuilds=true", + "pnpm store path", + } + if got := runner.ran(); strings.Join(got, "\n") != strings.Join(want, "\n") { + t.Fatalf("commands mismatch\n got: %v\nwant: %v", got, want) + } +} + +func TestWarmCacheWarmsEverySpec(t *testing.T) { + runner := &updateFakeRunner{succeedByDefault: true} + results, err := WarmCache(context.Background(), WarmOptions{ + Manager: ManagerGo, + Specs: []string{"github.com/acme/one@v1.0.0", "github.com/acme/two@v2.0.0"}, + Runner: runner, + }) + if err != nil { + t.Fatal(err) + } + if len(results) != 2 { + t.Fatalf("want 2 results, got %d", len(results)) + } + // Specs are warmed concurrently, so assert identity by position rather than + // relying on command ordering. + if results[0].Spec != "github.com/acme/one@v1.0.0" || results[1].Spec != "github.com/acme/two@v2.0.0" { + t.Fatalf("results should stay in spec order, got %q and %q", results[0].Spec, results[1].Spec) + } + for _, result := range results { + if result.Error != "" { + t.Errorf("spec %q failed: %s", result.Spec, result.Error) + } + } +} + +// A bad spec must be reported against that spec rather than aborting the run +// before the others are warmed. +func TestWarmCacheReportsABadSpecWithoutSkippingTheRest(t *testing.T) { + runner := &updateFakeRunner{succeedByDefault: true} + results, err := WarmCache(context.Background(), WarmOptions{ + Manager: ManagerGo, + Specs: []string{"left-pad@", "github.com/acme/lib@v1.0.0"}, + Runner: runner, + }) + if err == nil { + t.Fatal("expected a non-nil error because one spec is invalid") + } + if results[0].Error == "" { + t.Error("the invalid spec should carry an error") + } + if results[1].Error != "" { + t.Errorf("the valid spec should still have been warmed, got %q", results[1].Error) + } +} diff --git a/deps/chart_remote.go b/deps/chart_remote.go new file mode 100644 index 0000000..cfbafb0 --- /dev/null +++ b/deps/chart_remote.go @@ -0,0 +1,43 @@ +package deps + +import ( + "context" + "fmt" +) + +// chartResolver fetches a Helm chart dependency and produces its direct children: +// nested subchart dependency nodes and image nodes harvested from the fetched +// chart's values.yaml and templates. +type chartResolver struct { + helm *helmClient +} + +func (c *chartResolver) expand(ctx context.Context, dep chartDepEntry, parentDepth int) (children []*Node, resolvedVersion string, warnings []string) { + fc, cleanup, err := c.helm.fetchChart(ctx, dep) + if err != nil { + return nil, "", []string{fmt.Sprintf("chart %s@%s: %s", dep.Name, dep.Version, err)} + } + defer cleanup() + + for _, sub := range fc.Dependencies { + if sub.Name == "" { + continue + } + n := NewNode(ManagerHelm, sub.Name, sub.Version) + n.Depth = parentDepth + 1 + n.Scope = "dependencies" + n.Source = sub.Repository + children = append(children, n) + } + + imgNodes, imgWarns := chartImageNodes(fc.Dir, fc.Dir) + for _, img := range imgNodes { + img.Depth = parentDepth + 1 + img.Direct = false + children = append(children, img) + } + for _, w := range imgWarns { + warnings = append(warnings, w.Message) + } + return children, fc.Version, warnings +} diff --git a/deps/collapse.go b/deps/collapse.go new file mode 100644 index 0000000..1fb8620 --- /dev/null +++ b/deps/collapse.go @@ -0,0 +1,55 @@ +package deps + +// collapseDuplicates rewrites package-manager roots in place so each dependency +// renders once at its resolved (shallowest) location instead of repeating under +// every parent. Later occurrences are dropped silently with no marker. Image/Helm +// roots are left untouched: repeated images are real distinct deployments. +func collapseDuplicates(roots []*Node) { + for _, root := range roots { + if root == nil || !isPackageManager(root.Manager) { + continue + } + collapseRoot(root) + } +} + +func isPackageManager(manager Manager) bool { + switch manager { + case ManagerGo, ManagerMaven, ManagerGradle, ManagerNPM, ManagerPNPM: + return true + } + return false +} + +// collapseRoot keeps the first BFS sighting (shallowest, then sorted) of each +// node ID within a single root and silently drops later sightings. +func collapseRoot(root *Node) { + sortTree(root) + + seen := map[string]bool{} + queue := []*Node{root} + for len(queue) > 0 { + parent := queue[0] + queue = queue[1:] + kept := parent.Children[:0] + for _, child := range parent.Children { + if seen[child.ID] { + continue + } + seen[child.ID] = true + kept = append(kept, child) + queue = append(queue, child) + } + parent.Children = kept + } +} + +func sortTree(node *Node) { + if node == nil { + return + } + sortChildren(node) + for _, child := range node.Children { + sortTree(child) + } +} diff --git a/deps/collapse_test.go b/deps/collapse_test.go new file mode 100644 index 0000000..a9793ec --- /dev/null +++ b/deps/collapse_test.go @@ -0,0 +1,111 @@ +package deps + +import "testing" + +// diamondRoot builds a go project where a shared dependency is reachable under +// three direct parents, the classic diamond that motivates collapsing. +// +// app +// ├── a ── shared@v1 +// ├── b ── shared@v1 +// └── c ── shared@v1 +func diamondRoot() *Node { + root := NewNode(ManagerGo, "github.com/acme/app", "") + root.Source = "go.mod" + shared := func() *Node { + n := NewNode(ManagerGo, "github.com/acme/shared", "v1.0.0") + n.Depth = 2 + return n + } + for _, name := range []string{"a", "b", "c"} { + parent := NewNode(ManagerGo, "github.com/acme/"+name, "v1.0.0") + parent.Direct = true + parent.Depth = 1 + parent.Children = []*Node{shared()} + root.Children = append(root.Children, parent) + } + return root +} + +func TestCollapseKeepsSharedDependencyOnce(t *testing.T) { + root := diamondRoot() + collapseDuplicates([]*Node{root}) + + var kept int + for _, parent := range root.Children { + if findChild(parent, "github.com/acme/shared") != nil { + kept++ + } + } + if kept != 1 { + t.Fatalf("expected shared dependency to survive under exactly one parent, got %d", kept) + } +} + +func TestCollapseDropsLaterSightingsSilently(t *testing.T) { + root := diamondRoot() + collapseDuplicates([]*Node{root}) + + a := findChild(root, "github.com/acme/a") + if findChild(a, "github.com/acme/shared") == nil { + t.Fatalf("shared should be retained under the first sorted parent 'a'") + } + for _, name := range []string{"github.com/acme/b", "github.com/acme/c"} { + parent := findChild(root, name) + if findChild(parent, "github.com/acme/shared") != nil { + t.Fatalf("parent %s should no longer carry the shared dependency", name) + } + if len(parent.Children) != 0 { + t.Fatalf("parent %s should have no children after collapse, got %d", name, len(parent.Children)) + } + } +} + +func TestCollapseLeavesImageRootsUntouched(t *testing.T) { + root := NewNode(ManagerImage, "container images", "") + dup := func() *Node { + n := NewNode(ManagerImage, "nginx", "1.25") + n.Depth = 1 + return n + } + root.Children = []*Node{dup(), dup()} + collapseDuplicates([]*Node{root}) + + if len(root.Children) != 2 { + t.Fatalf("image roots must keep every occurrence, got %d children", len(root.Children)) + } +} + +func TestCollapseIsolatesPerRoot(t *testing.T) { + first := diamondRoot() + second := diamondRoot() + collapseDuplicates([]*Node{first, second}) + + for _, root := range []*Node{first, second} { + a := findChild(root, "github.com/acme/a") + if findChild(a, "github.com/acme/shared") == nil { + t.Fatalf("each root should retain shared under its first sorted parent") + } + for _, name := range []string{"github.com/acme/b", "github.com/acme/c"} { + if findChild(findChild(root, name), "github.com/acme/shared") != nil { + t.Fatalf("each root should dedup independently; %s should not keep shared", name) + } + } + } +} + +func TestCollapseKeepsConflictingVersions(t *testing.T) { + root := NewNode(ManagerGo, "github.com/acme/app", "") + v1 := NewNode(ManagerGo, "github.com/acme/lib", "v1.0.0") + v1.Direct = true + v1.Depth = 1 + v2 := NewNode(ManagerGo, "github.com/acme/lib", "v2.0.0") + v2.Direct = true + v2.Depth = 1 + root.Children = []*Node{v1, v2} + collapseDuplicates([]*Node{root}) + + if len(root.Children) != 2 { + t.Fatalf("conflicting versions must each survive, got %d", len(root.Children)) + } +} diff --git a/deps/common.go b/deps/common.go new file mode 100644 index 0000000..1412e25 --- /dev/null +++ b/deps/common.go @@ -0,0 +1,72 @@ +package deps + +import ( + "path/filepath" + "sort" + "strings" +) + +func sortChildren(node *Node) { + if node == nil { + return + } + sort.SliceStable(node.Children, func(i, j int) bool { + return dependencyLess(node.Children[i], node.Children[j]) + }) +} + +func dependencyLess(a, b *Node) bool { + if a == nil { + return b != nil + } + if b == nil { + return false + } + if ar, br := dependencySortRank(a), dependencySortRank(b); ar != br { + return ar < br + } + if an, bn := strings.ToLower(a.Name), strings.ToLower(b.Name); an != bn { + return an < bn + } + if a.Name != b.Name { + return a.Name < b.Name + } + if a.Version != b.Version { + return a.Version < b.Version + } + if a.Manager != b.Manager { + return a.Manager < b.Manager + } + return a.ID < b.ID +} + +func dependencySortRank(node *Node) int { + if isReplacementDependency(node) { + return 0 + } + if node != nil && node.Direct { + return 1 + } + return 2 +} + +func isReplacementDependency(node *Node) bool { + if node == nil { + return false + } + if node.Local { + return true + } + return node.Manager == ManagerGo && node.Source != "" && node.Source != "go.mod" +} + +func isLocalRef(ref string) bool { + ref = strings.TrimSpace(ref) + if ref == "" { + return false + } + if strings.HasPrefix(ref, "file:") || strings.HasPrefix(ref, "link:") || strings.HasPrefix(ref, "portal:") { + return true + } + return strings.HasPrefix(ref, ".") || strings.HasPrefix(ref, "/") || strings.HasPrefix(ref, ".."+string(filepath.Separator)) || strings.HasPrefix(ref, "../") +} diff --git a/deps/compare.go b/deps/compare.go new file mode 100644 index 0000000..8dcfa55 --- /dev/null +++ b/deps/compare.go @@ -0,0 +1,243 @@ +package deps + +import ( + "path/filepath" + "sort" + "strings" +) + +type ChangeType string + +const ( + ChangeAdded ChangeType = "added" + ChangeRemoved ChangeType = "removed" + ChangeUpdated ChangeType = "updated" +) + +// Change describes a single dependency difference between two scans, scoped to a +// project (manifest) within the scanned tree. +type Change struct { + Type ChangeType `json:"type"` + Manager Manager `json:"manager"` + Name string `json:"name"` + Project string `json:"project"` + OldVersion string `json:"old_version,omitempty"` + NewVersion string `json:"new_version,omitempty"` + OldScope string `json:"old_scope,omitempty"` + NewScope string `json:"new_scope,omitempty"` + Direct bool `json:"direct,omitempty"` + Depth int `json:"depth,omitempty"` +} + +type ComparisonMetadata struct { + Path string `json:"path"` + BaseRef string `json:"base_ref"` + HeadRef string `json:"head_ref"` +} + +type ComparisonStatistics struct { + Added int `json:"added"` + Removed int `json:"removed"` + Updated int `json:"updated"` +} + +// Comparison is the result of diffing two dependency graphs. +type Comparison struct { + Metadata ComparisonMetadata `json:"metadata"` + Added []Change `json:"added,omitempty"` + Removed []Change `json:"removed,omitempty"` + Updated []Change `json:"updated,omitempty"` + Statistics ComparisonStatistics `json:"statistics"` + Warnings []Warning `json:"warnings,omitempty"` +} + +type depEntry struct { + manager Manager + name string + versions map[string]bool + scope string + direct bool + depth int +} + +// Compare diffs the dependency graphs of two exports. Dependencies are keyed by +// manager+name within each project (manifest), so the same package in different +// projects is compared independently. Duplicate occurrences of a package within +// a project are collapsed into a joined version set. +func Compare(base, head *Export) *Comparison { + baseIndex := indexExport(base) + headIndex := indexExport(head) + + comparison := &Comparison{} + for _, project := range sortedKeys(unionKeys(baseIndex, headIndex)) { + baseDeps := baseIndex[project] + headDeps := headIndex[project] + for _, key := range sortedKeys(unionKeys(baseDeps, headDeps)) { + before, hasBefore := baseDeps[key] + after, hasAfter := headDeps[key] + switch { + case hasBefore && !hasAfter: + comparison.Removed = append(comparison.Removed, removedChange(project, before)) + case !hasBefore && hasAfter: + comparison.Added = append(comparison.Added, addedChange(project, after)) + default: + if change, ok := updatedChange(project, before, after); ok { + comparison.Updated = append(comparison.Updated, change) + } + } + } + } + comparison.Statistics = ComparisonStatistics{ + Added: len(comparison.Added), + Removed: len(comparison.Removed), + Updated: len(comparison.Updated), + } + return comparison +} + +func addedChange(project string, e *depEntry) Change { + return Change{ + Type: ChangeAdded, + Manager: e.manager, + Name: e.name, + Project: project, + NewVersion: versionSetString(e.versions), + NewScope: e.scope, + Direct: e.direct, + Depth: e.depth, + } +} + +func removedChange(project string, e *depEntry) Change { + return Change{ + Type: ChangeRemoved, + Manager: e.manager, + Name: e.name, + Project: project, + OldVersion: versionSetString(e.versions), + OldScope: e.scope, + Direct: e.direct, + Depth: e.depth, + } +} + +func updatedChange(project string, before, after *depEntry) (Change, bool) { + oldVersion := versionSetString(before.versions) + newVersion := versionSetString(after.versions) + if oldVersion == newVersion && before.scope == after.scope { + return Change{}, false + } + return Change{ + Type: ChangeUpdated, + Manager: after.manager, + Name: after.name, + Project: project, + OldVersion: oldVersion, + NewVersion: newVersion, + OldScope: before.scope, + NewScope: after.scope, + Direct: after.direct, + Depth: after.depth, + }, true +} + +func indexExport(export *Export) map[string]map[string]*depEntry { + out := map[string]map[string]*depEntry{} + if export == nil { + return out + } + for _, root := range export.Roots { + project := normalizeProject(export.Metadata.Path, root) + deps := out[project] + if deps == nil { + deps = map[string]*depEntry{} + out[project] = deps + } + for _, child := range root.Children { + collectEntries(child, deps) + } + } + return out +} + +func collectEntries(node *Node, deps map[string]*depEntry) { + if node == nil { + return + } + key := string(node.Manager) + ":" + node.Name + entry := deps[key] + if entry == nil { + entry = &depEntry{ + manager: node.Manager, + name: node.Name, + versions: map[string]bool{}, + scope: node.Scope, + direct: node.Direct, + depth: node.Depth, + } + deps[key] = entry + } else { + if node.Depth < entry.depth { + entry.depth = node.Depth + } + if node.Direct { + entry.direct = true + } + if entry.scope == "" { + entry.scope = node.Scope + } + } + if node.Version != "" { + entry.versions[node.Version] = true + } + for _, child := range node.Children { + collectEntries(child, deps) + } +} + +func normalizeProject(basePath string, root *Node) string { + target := root.Path + if target == "" { + return root.Name + } + if basePath == "" { + return filepath.ToSlash(target) + } + rel, err := filepath.Rel(basePath, target) + if err != nil { + return filepath.ToSlash(target) + } + return filepath.ToSlash(rel) +} + +func versionSetString(versions map[string]bool) string { + if len(versions) == 0 { + return "" + } + list := make([]string, 0, len(versions)) + for version := range versions { + list = append(list, version) + } + sort.Strings(list) + return strings.Join(list, ", ") +} + +func unionKeys[V any](a, b map[string]V) map[string]bool { + keys := map[string]bool{} + for key := range a { + keys[key] = true + } + for key := range b { + keys[key] = true + } + return keys +} + +func sortedKeys(set map[string]bool) []string { + keys := make([]string, 0, len(set)) + for key := range set { + keys = append(keys, key) + } + sort.Strings(keys) + return keys +} diff --git a/deps/compare_pretty.go b/deps/compare_pretty.go new file mode 100644 index 0000000..fd6cc27 --- /dev/null +++ b/deps/compare_pretty.go @@ -0,0 +1,168 @@ +package deps + +import ( + "fmt" + "sort" + + "github.com/flanksource/clicky" + "github.com/flanksource/clicky/api" +) + +// diffNode adapts a comparison group (project header or change leaf) to the +// clicky tree renderer. +type diffNode struct { + text api.Text + children []api.TreeNode +} + +var _ api.TreeNode = (*diffNode)(nil) + +func (d *diffNode) Pretty() api.Text { return d.text } + +func (d *diffNode) GetChildren() []api.TreeNode { return d.children } + +func (c *Comparison) Pretty() api.Text { + if c == nil { + return clicky.Text("") + } + t := clicky.Text("Dependency Diff", "font-bold") + t = t.Append(summaryText(c.Statistics)) + if c.Metadata.BaseRef != "" { + t = t.Append(fmt.Sprintf(" %s..%s", c.Metadata.BaseRef, headRefLabel(c.Metadata.HeadRef)), "text-muted") + } + + nodes := diffProjectNodes(c) + if len(nodes) == 0 { + return t.NewLine().Append("No dependency changes", "text-muted") + } + t = t.NewLine().Add(api.NewTree(nodes...)) + + if len(c.Warnings) > 0 { + t = t.NewLine().Append("Warnings", "font-bold text-yellow-600") + for _, w := range c.Warnings { + t = t.NewLine().Append("- "+w.Message, "text-yellow-600") + } + } + return t +} + +func summaryText(stats ComparisonStatistics) api.Text { + return clicky.Text(" +", "text-muted"). + Append(fmt.Sprintf("%d", stats.Added), "text-green-600"). + Append(" -", "text-muted"). + Append(fmt.Sprintf("%d", stats.Removed), "text-red-600"). + Append(" ~", "text-muted"). + Append(fmt.Sprintf("%d", stats.Updated), "text-yellow-600") +} + +func headRefLabel(headRef string) string { + if headRef == "" { + return "working tree" + } + return headRef +} + +func diffProjectNodes(c *Comparison) []api.TreeNode { + byProject := map[string][]Change{} + for _, change := range allChanges(c) { + byProject[change.Project] = append(byProject[change.Project], change) + } + projects := make([]string, 0, len(byProject)) + for project := range byProject { + projects = append(projects, project) + } + sort.Strings(projects) + + nodes := make([]api.TreeNode, 0, len(projects)) + for _, project := range projects { + changes := byProject[project] + sort.SliceStable(changes, func(i, j int) bool { + if changes[i].Manager != changes[j].Manager { + return changes[i].Manager < changes[j].Manager + } + return changes[i].Name < changes[j].Name + }) + projectNode := &diffNode{text: clicky.Text(project, "font-bold")} + for _, change := range changes { + projectNode.children = append(projectNode.children, &diffNode{text: changeText(change)}) + } + nodes = append(nodes, projectNode) + } + return nodes +} + +func allChanges(c *Comparison) []Change { + out := make([]Change, 0, len(c.Added)+len(c.Removed)+len(c.Updated)) + out = append(out, c.Added...) + out = append(out, c.Removed...) + out = append(out, c.Updated...) + return out +} + +func changeText(change Change) api.Text { + t := clicky.Text("[", "text-muted"). + Append(string(change.Manager), managerStyle(change.Manager)). + Append("] ", "text-muted") + switch change.Type { + case ChangeAdded: + t = t.Append(change.Name, "font-bold text-green-600") + if change.NewVersion != "" { + t = t.Append("@"+change.NewVersion, "font-mono text-green-600") + } + t = t.Space().Append("(new)", "text-green-600") + case ChangeRemoved: + t = t.Append(change.Name, "font-bold line-through text-red-600") + if change.OldVersion != "" { + t = t.Append("@"+change.OldVersion, "font-mono line-through text-red-600") + } + t = t.Space().Append("(removed)", "text-red-600") + case ChangeUpdated: + t = t.Append(change.Name, "font-bold text-cyan-600"). + Space().Append(versionOrNone(change.OldVersion), "font-mono text-muted"). + Append(" → ", "text-yellow-600"). + Append(versionOrNone(change.NewVersion), "font-mono text-yellow-600") + if change.OldScope != change.NewScope { + t = t.Space().Append(fmt.Sprintf("(%s → %s)", change.OldScope, change.NewScope), "text-muted") + } + } + return t +} + +func versionOrNone(version string) string { + if version == "" { + return "(none)" + } + return version +} + +// Columns implements the clicky TableProvider interface for flat change tables. +func (Change) Columns() []api.ColumnDef { + return []api.ColumnDef{ + api.Column("type").Label("Type").Build(), + api.Column("manager").Label("Manager").Build(), + api.Column("dependency").Label("Dependency").Build(), + api.Column("project").Label("Project").Build(), + api.Column("change").Label("Change").Build(), + } +} + +func (change Change) Row() map[string]any { + return map[string]any{ + "type": clicky.Text(string(change.Type), changeTypeStyle(change.Type)), + "manager": clicky.Text(string(change.Manager), managerStyle(change.Manager)), + "dependency": clicky.Text(change.Name, "font-bold text-cyan-600"), + "project": clicky.Text(change.Project, "font-mono"), + "change": changeText(change), + } +} + +func changeTypeStyle(changeType ChangeType) string { + switch changeType { + case ChangeAdded: + return "text-green-600" + case ChangeRemoved: + return "text-red-600" + default: + return "text-yellow-600" + } +} diff --git a/deps/compare_pretty_test.go b/deps/compare_pretty_test.go new file mode 100644 index 0000000..c90692d --- /dev/null +++ b/deps/compare_pretty_test.go @@ -0,0 +1,60 @@ +package deps + +import ( + "encoding/json" + "strings" + "testing" +) + +func sampleComparison() *Comparison { + return &Comparison{ + Metadata: ComparisonMetadata{Path: "/repo", BaseRef: "HEAD~1", HeadRef: ""}, + Added: []Change{{Type: ChangeAdded, Manager: ManagerGo, Name: "fresh", Project: "svc/go.mod", NewVersion: "0.1"}}, + Removed: []Change{{Type: ChangeRemoved, Manager: ManagerGo, Name: "gone", Project: "svc/go.mod", OldVersion: "1.0"}}, + Updated: []Change{{Type: ChangeUpdated, Manager: ManagerGo, Name: "bump", Project: "svc/go.mod", OldVersion: "1.0", NewVersion: "2.0"}}, + Statistics: ComparisonStatistics{Added: 1, Removed: 1, Updated: 1}, + } +} + +func TestComparisonPrettyMarkers(t *testing.T) { + out := sampleComparison().Pretty().String() + for _, want := range []string{"+1", "-1", "~1", "svc/go.mod", "fresh", "(new)", "gone", "(removed)", "bump", "→", "2.0", "working tree"} { + if !strings.Contains(out, want) { + t.Fatalf("pretty output missing %q:\n%s", want, out) + } + } +} + +func TestComparisonPrettyEmpty(t *testing.T) { + out := (&Comparison{}).Pretty().String() + if !strings.Contains(out, "No dependency changes") { + t.Fatalf("empty comparison should report no changes, got %q", out) + } +} + +func TestComparisonPrettyPrunesToChangedProjects(t *testing.T) { + c := sampleComparison() + c.Added = append(c.Added, Change{Type: ChangeAdded, Manager: ManagerNPM, Name: "left-pad", Project: "web/package.json", NewVersion: "1.3.0"}) + out := c.Pretty().String() + if !strings.Contains(out, "web/package.json") || !strings.Contains(out, "svc/go.mod") { + t.Fatalf("both changed projects should appear:\n%s", out) + } +} + +func TestComparisonJSONShape(t *testing.T) { + data, err := json.Marshal(sampleComparison()) + if err != nil { + t.Fatal(err) + } + body := string(data) + for _, want := range []string{`"added"`, `"removed"`, `"updated"`, `"statistics"`} { + if !strings.Contains(body, want) { + t.Fatalf("json missing %q: %s", want, body) + } + } + for _, unwanted := range []string{`"roots"`, `"nodes"`, `"children"`} { + if strings.Contains(body, unwanted) { + t.Fatalf("json should not leak tree field %q: %s", unwanted, body) + } + } +} diff --git a/deps/compare_scan.go b/deps/compare_scan.go new file mode 100644 index 0000000..4169583 --- /dev/null +++ b/deps/compare_scan.go @@ -0,0 +1,129 @@ +package deps + +import ( + "bytes" + "context" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + + "github.com/flanksource/repomap" +) + +// CompareOptions configures a dependency diff. BaseRef is required; HeadRef may +// be empty to compare against the current working tree. +type CompareOptions struct { + Options + BaseRef string + HeadRef string +} + +// CompareScan resolves dependency graphs at two git revisions (or a revision and +// the working tree) and diffs them. Each non-working-tree side is materialized +// with `git worktree add --detach` so image/Helm discovery, which requires a +// real checkout, works unchanged. A dirty working tree is not an error. +func CompareScan(ctx context.Context, path string, opts CompareOptions) (*Comparison, error) { + if opts.BaseRef == "" { + return nil, fmt.Errorf("base git ref is required") + } + absPath, err := filepath.Abs(path) + if err != nil { + return nil, err + } + repoRoot := repomap.FindGitRoot(absPath) + if repoRoot == "" { + return nil, fmt.Errorf("not a git repository: %s", absPath) + } + relPath, err := filepath.Rel(repoRoot, absPath) + if err != nil { + return nil, err + } + + // Diffs always compare the full graph: indexExport walks Roots, so collapsing + // duplicate subtrees would hide real dependency changes. + opts.ShowDuplicates = true + + baseSha, err := resolveRef(ctx, repoRoot, opts.BaseRef) + if err != nil { + return nil, err + } + baseExport, err := scanAtRef(ctx, repoRoot, baseSha, relPath, opts.Options) + if err != nil { + return nil, err + } + + var headExport *Export + if opts.HeadRef == "" { + headExport, err = Scan(ctx, absPath, opts.Options) + if err != nil { + return nil, err + } + } else { + headSha, refErr := resolveRef(ctx, repoRoot, opts.HeadRef) + if refErr != nil { + return nil, refErr + } + headExport, err = scanAtRef(ctx, repoRoot, headSha, relPath, opts.Options) + if err != nil { + return nil, err + } + } + + comparison := Compare(baseExport, headExport) + comparison.Metadata = ComparisonMetadata{Path: absPath, BaseRef: opts.BaseRef, HeadRef: opts.HeadRef} + return comparison, nil +} + +func scanAtRef(ctx context.Context, repoRoot, sha, relPath string, opts Options) (*Export, error) { + worktree, cleanup, err := addWorktree(ctx, repoRoot, sha) + if err != nil { + return nil, err + } + defer cleanup() + scanDir := filepath.Join(worktree, relPath) + if _, statErr := os.Stat(scanDir); statErr != nil { + return nil, fmt.Errorf("path %q does not exist at ref %s: %w", relPath, sha, statErr) + } + return Scan(ctx, scanDir, opts) +} + +func addWorktree(ctx context.Context, repoRoot, sha string) (string, func(), error) { + parent, err := os.MkdirTemp("", "repomap-diff-*") + if err != nil { + return "", nil, err + } + worktree := filepath.Join(parent, "worktree") + if _, err := runGitCmd(ctx, repoRoot, "worktree", "add", "--detach", worktree, sha); err != nil { + _ = os.RemoveAll(parent) + return "", nil, err + } + cleanup := func() { + _, _ = runGitCmd(ctx, repoRoot, "worktree", "remove", "--force", worktree) + _ = os.RemoveAll(parent) + } + return worktree, cleanup, nil +} + +func resolveRef(ctx context.Context, repoRoot, ref string) (string, error) { + out, err := runGitCmd(ctx, repoRoot, "rev-parse", "--verify", ref+"^{commit}") + if err != nil { + return "", fmt.Errorf("invalid git ref %q: %w", ref, err) + } + return strings.TrimSpace(out), nil +} + +func runGitCmd(ctx context.Context, dir string, args ...string) (string, error) { + cmd := exec.CommandContext(ctx, "git", append([]string{"-C", dir}, args...)...) + var stdout, stderr bytes.Buffer + cmd.Stdout = &stdout + cmd.Stderr = &stderr + if err := cmd.Run(); err != nil { + if msg := strings.TrimSpace(stderr.String()); msg != "" { + return "", fmt.Errorf("git %s: %s: %w", strings.Join(args, " "), msg, err) + } + return "", fmt.Errorf("git %s: %w", strings.Join(args, " "), err) + } + return stdout.String(), nil +} diff --git a/deps/compare_scan_test.go b/deps/compare_scan_test.go new file mode 100644 index 0000000..f33bd1c --- /dev/null +++ b/deps/compare_scan_test.go @@ -0,0 +1,123 @@ +package deps + +import ( + "context" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +func initRepo(t *testing.T) string { + t.Helper() + if _, err := exec.LookPath("git"); err != nil { + t.Skip("git not available") + } + dir := t.TempDir() + runGit(t, dir, "init") + runGit(t, dir, "config", "user.email", "test@example.com") + runGit(t, dir, "config", "user.name", "test") + return dir +} + +func gitOutput(t *testing.T, dir string, args ...string) string { + t.Helper() + cmd := exec.Command("git", args...) + cmd.Dir = dir + out, err := cmd.Output() + if err != nil { + t.Fatalf("git %s failed: %v", strings.Join(args, " "), err) + } + return string(out) +} + +func TestCompareScanHeadDiffAgainstWorkingTree(t *testing.T) { + dir := initRepo(t) + gomod := filepath.Join(dir, "go.mod") + writeFile(t, gomod, "module github.com/acme/app\n\ngo 1.22\n\nrequire github.com/acme/a v1.0.0\n") + runGit(t, dir, "add", ".") + runGit(t, dir, "commit", "-m", "base") + writeFile(t, gomod, "module github.com/acme/app\n\ngo 1.22\n\nrequire github.com/acme/a v1.5.0\n") + + cmp, err := CompareScan(context.Background(), dir, CompareOptions{Options: Options{MaxDepth: 1}, BaseRef: "HEAD"}) + if err != nil { + t.Fatal(err) + } + if len(cmp.Updated) != 1 || cmp.Updated[0].Name != "github.com/acme/a" || cmp.Updated[0].OldVersion != "v1.0.0" || cmp.Updated[0].NewVersion != "v1.5.0" { + t.Fatalf("expected a 1.0.0 -> 2.0.0, got %+v", cmp.Updated) + } + if lines := strings.TrimSpace(gitOutput(t, dir, "worktree", "list")); strings.Count(lines, "\n") != 0 { + t.Fatalf("worktree not cleaned up:\n%s", lines) + } +} + +func TestCompareScanRefRange(t *testing.T) { + dir := initRepo(t) + gomod := filepath.Join(dir, "go.mod") + writeFile(t, gomod, "module github.com/acme/app\n\ngo 1.22\n\nrequire github.com/acme/a v1.0.0\n") + runGit(t, dir, "add", ".") + runGit(t, dir, "commit", "-m", "v1") + writeFile(t, gomod, "module github.com/acme/app\n\ngo 1.22\n\nrequire github.com/acme/a v1.5.0\n") + runGit(t, dir, "add", ".") + runGit(t, dir, "commit", "-m", "v2") + + cmp, err := CompareScan(context.Background(), dir, CompareOptions{Options: Options{MaxDepth: 1}, BaseRef: "HEAD~1", HeadRef: "HEAD"}) + if err != nil { + t.Fatal(err) + } + if len(cmp.Updated) != 1 || cmp.Updated[0].NewVersion != "v1.5.0" { + t.Fatalf("ref1..ref2 diff failed: %+v", cmp.Updated) + } +} + +func TestCompareScanFilterAppliedBothSides(t *testing.T) { + dir := initRepo(t) + gomod := filepath.Join(dir, "go.mod") + writeFile(t, gomod, "module github.com/acme/app\n\ngo 1.22\n\nrequire (\n\tgithub.com/acme/alpha v1.0.0\n\tgithub.com/acme/beta v1.0.0\n)\n") + runGit(t, dir, "add", ".") + runGit(t, dir, "commit", "-m", "base") + writeFile(t, gomod, "module github.com/acme/app\n\ngo 1.22\n\nrequire (\n\tgithub.com/acme/alpha v1.5.0\n\tgithub.com/acme/beta v1.5.0\n)\n") + + cmp, err := CompareScan(context.Background(), dir, CompareOptions{Options: Options{MaxDepth: 1, Filters: []string{"*alpha*"}}, BaseRef: "HEAD"}) + if err != nil { + t.Fatal(err) + } + if len(cmp.Updated) != 1 || !strings.Contains(cmp.Updated[0].Name, "alpha") { + t.Fatalf("filter should restrict both sides to alpha, got %+v", cmp.Updated) + } +} + +func TestCompareScanInvalidRef(t *testing.T) { + dir := initRepo(t) + writeFile(t, filepath.Join(dir, "go.mod"), "module github.com/acme/app\n\ngo 1.22\n") + runGit(t, dir, "add", ".") + runGit(t, dir, "commit", "-m", "base") + + _, err := CompareScan(context.Background(), dir, CompareOptions{Options: Options{MaxDepth: 1}, BaseRef: "does-not-exist"}) + if err == nil || !strings.Contains(err.Error(), "invalid git ref") { + t.Fatalf("expected invalid ref error, got %v", err) + } +} + +func TestCompareScanNonGitDir(t *testing.T) { + dir := t.TempDir() + writeFile(t, filepath.Join(dir, "go.mod"), "module github.com/acme/app\n\ngo 1.22\n") + _, err := CompareScan(context.Background(), dir, CompareOptions{Options: Options{MaxDepth: 1}, BaseRef: "HEAD"}) + if err == nil || !strings.Contains(err.Error(), "not a git repository") { + t.Fatalf("expected non-git error, got %v", err) + } +} + +func TestCompareScanPathMissingAtRef(t *testing.T) { + dir := initRepo(t) + writeFile(t, filepath.Join(dir, "go.mod"), "module github.com/acme/app\n\ngo 1.22\n") + runGit(t, dir, "add", ".") + runGit(t, dir, "commit", "-m", "base") + // svc only exists in the working tree, not at HEAD. + writeFile(t, filepath.Join(dir, "svc", "go.mod"), "module github.com/acme/svc\n\ngo 1.22\n") + + _, err := CompareScan(context.Background(), filepath.Join(dir, "svc"), CompareOptions{Options: Options{MaxDepth: 1}, BaseRef: "HEAD"}) + if err == nil || !strings.Contains(err.Error(), "does not exist at ref") { + t.Fatalf("expected path-missing-at-ref error, got %v", err) + } +} diff --git a/deps/compare_test.go b/deps/compare_test.go new file mode 100644 index 0000000..b2324cd --- /dev/null +++ b/deps/compare_test.go @@ -0,0 +1,139 @@ +package deps + +import ( + "path/filepath" + "testing" +) + +func goDep(name, version, scope string) *Node { + n := NewNode(ManagerGo, name, version) + n.Scope = scope + n.Depth = 1 + n.Direct = true + return n +} + +func goProject(metaPath, modRel string, children ...*Node) *Node { + root := NewNode(ManagerGo, "github.com/acme/"+modRel, "") + root.Path = filepath.Join(metaPath, modRel, "go.mod") + root.Children = children + return root +} + +func exportWith(metaPath string, roots ...*Node) *Export { + return &Export{Metadata: Metadata{Path: metaPath}, Roots: roots} +} + +func totalChanges(c *Comparison) int { + return len(c.Added) + len(c.Removed) + len(c.Updated) +} + +func TestCompareIdenticalIsEmpty(t *testing.T) { + base := exportWith("/repo", goProject("/repo", "svc", goDep("a", "1.0", "require"))) + head := exportWith("/repo", goProject("/repo", "svc", goDep("a", "1.0", "require"))) + got := Compare(base, head) + if totalChanges(got) != 0 { + t.Fatalf("identical graphs should produce no changes, got %+v", got) + } +} + +func TestCompareAddRemoveUpdate(t *testing.T) { + base := exportWith("/repo", goProject("/repo", "svc", + goDep("keep", "1.0", "require"), + goDep("bump", "1.0", "require"), + goDep("gone", "1.0", "require"), + )) + head := exportWith("/repo", goProject("/repo", "svc", + goDep("keep", "1.0", "require"), + goDep("bump", "2.0", "require"), + goDep("fresh", "0.1", "require"), + )) + got := Compare(base, head) + if len(got.Added) != 1 || got.Added[0].Name != "fresh" || got.Added[0].NewVersion != "0.1" { + t.Fatalf("added = %+v", got.Added) + } + if len(got.Removed) != 1 || got.Removed[0].Name != "gone" { + t.Fatalf("removed = %+v", got.Removed) + } + if len(got.Updated) != 1 || got.Updated[0].OldVersion != "1.0" || got.Updated[0].NewVersion != "2.0" { + t.Fatalf("updated = %+v", got.Updated) + } + if got.Updated[0].Project != "svc/go.mod" { + t.Fatalf("project key = %q, want svc/go.mod", got.Updated[0].Project) + } +} + +func TestCompareScopeOnlyChangeIsUpdate(t *testing.T) { + base := exportWith("/repo", goProject("/repo", "svc", goDep("a", "1.0", "require"))) + head := exportWith("/repo", goProject("/repo", "svc", goDep("a", "1.0", "indirect"))) + got := Compare(base, head) + if len(got.Updated) != 1 || got.Updated[0].OldScope != "require" || got.Updated[0].NewScope != "indirect" { + t.Fatalf("scope-only update not detected: %+v", got.Updated) + } +} + +func TestComparePerProjectKeying(t *testing.T) { + base := exportWith("/repo", + goProject("/repo", "a", goDep("lib", "1.0", "require")), + goProject("/repo", "b", goDep("lib", "1.0", "require")), + ) + head := exportWith("/repo", + goProject("/repo", "a", goDep("lib", "2.0", "require")), + goProject("/repo", "b", goDep("lib", "1.0", "require")), + ) + got := Compare(base, head) + if len(got.Updated) != 1 || got.Updated[0].Project != "a/go.mod" { + t.Fatalf("expected only project a to change, got %+v", got.Updated) + } +} + +func TestCompareDuplicateVersionSets(t *testing.T) { + dupA := goDep("lib", "1.0", "require") + dupB := goDep("lib", "2.0", "require") + base := exportWith("/repo", goProject("/repo", "svc", dupA, dupB)) + head := exportWith("/repo", goProject("/repo", "svc", goDep("lib", "2.0", "require"))) + got := Compare(base, head) + if len(got.Updated) != 1 || got.Updated[0].OldVersion != "1.0, 2.0" || got.Updated[0].NewVersion != "2.0" { + t.Fatalf("duplicate version set not joined: %+v", got.Updated) + } +} + +func TestComparePathNormalization(t *testing.T) { + base := exportWith("/repo/base", goProject("/repo/base", "svc", goDep("a", "1.0", "require"))) + head := exportWith("/repo/head", goProject("/repo/head", "svc", goDep("a", "2.0", "require"))) + got := Compare(base, head) + if len(got.Updated) != 1 || got.Updated[0].Project != "svc/go.mod" { + t.Fatalf("absolute worktree paths not normalized to relative project key: %+v", got.Updated) + } +} + +func TestCompareWholeProjectAddRemove(t *testing.T) { + base := exportWith("/repo", goProject("/repo", "old", goDep("a", "1.0", "require"))) + head := exportWith("/repo", goProject("/repo", "new", goDep("b", "1.0", "require"))) + got := Compare(base, head) + if len(got.Removed) != 1 || got.Removed[0].Project != "old/go.mod" { + t.Fatalf("removed project deps = %+v", got.Removed) + } + if len(got.Added) != 1 || got.Added[0].Project != "new/go.mod" { + t.Fatalf("added project deps = %+v", got.Added) + } +} + +func TestCompareDeterministicOrdering(t *testing.T) { + base := exportWith("/repo", goProject("/repo", "svc")) + head := exportWith("/repo", goProject("/repo", "svc", + goDep("zebra", "1.0", "require"), + goDep("alpha", "1.0", "require"), + goDep("mango", "1.0", "require"), + )) + got := Compare(base, head) + want := []string{"alpha", "mango", "zebra"} + if len(got.Added) != 3 { + t.Fatalf("added = %d, want 3", len(got.Added)) + } + for i, name := range want { + if got.Added[i].Name != name { + t.Fatalf("added[%d] = %q, want %q (sorted)", i, got.Added[i].Name, name) + } + } +} diff --git a/deps/discover.go b/deps/discover.go new file mode 100644 index 0000000..0286635 --- /dev/null +++ b/deps/discover.go @@ -0,0 +1,208 @@ +package deps + +import ( + "fmt" + "io/fs" + "os" + osexec "os/exec" + "path/filepath" + "sort" + "strings" +) + +var ignoredDirs = map[string]bool{ + ".git": true, + ".gradle": true, + "build": true, + "dist": true, + "node_modules": true, + "target": true, + "vendor": true, +} + +func Discover(root string, managers []Manager) ([]Project, []Warning, error) { + return discover(root, managers, true) +} + +func discoverOffline(root string, managers []Manager) ([]Project, []Warning, error) { + return discover(root, managers, false) +} + +func discover(root string, managers []Manager, useGit bool) ([]Project, []Warning, error) { + selected := managerSet(managers) + absRoot, err := filepath.Abs(root) + if err != nil { + return nil, nil, err + } + info, err := os.Stat(absRoot) + if err != nil { + return nil, nil, err + } + if !info.IsDir() { + absRoot = filepath.Dir(absRoot) + } + + files, err := discoverManifestFiles(absRoot, useGit) + if err != nil { + return nil, nil, err + } + + byDir := map[string]map[string]string{} + for _, path := range files { + name := filepath.Base(path) + manager := managerForManifest(name) + if manager == "" { + continue + } + if len(selected) > 0 && !selected[manager] { + continue + } + dir := filepath.Dir(path) + if byDir[dir] == nil { + byDir[dir] = map[string]string{} + } + byDir[dir][name] = path + } + + var projects []Project + var warnings []Warning + for dir, files := range byDir { + if path := files["go.work"]; path != "" { + projects = append(projects, Project{Manager: ManagerGo, Dir: dir, File: path, Name: filepath.Base(dir)}) + } else if path := files["go.mod"]; path != "" { + projects = append(projects, Project{Manager: ManagerGo, Dir: dir, File: path, Name: filepath.Base(dir)}) + } + if path := files["pom.xml"]; path != "" { + projects = append(projects, Project{Manager: ManagerMaven, Dir: dir, File: path, Name: filepath.Base(dir)}) + } + if path := files["build.gradle"]; path != "" { + projects = append(projects, Project{Manager: ManagerGradle, Dir: dir, File: path, Name: filepath.Base(dir)}) + } else if path := files["build.gradle.kts"]; path != "" { + projects = append(projects, Project{Manager: ManagerGradle, Dir: dir, File: path, Name: filepath.Base(dir)}) + } + pnpmLock := files["pnpm-lock.yaml"] + npmLock := files["package-lock.json"] + shrinkwrap := files["npm-shrinkwrap.json"] + if pnpmLock != "" { + projects = append(projects, Project{Manager: ManagerPNPM, Dir: dir, File: pnpmLock, Name: filepath.Base(dir)}) + if npmLock != "" || shrinkwrap != "" { + warnings = append(warnings, Warning{ + Manager: ManagerPNPM, + Project: dir, + Message: "pnpm-lock.yaml and npm lockfile both found; using pnpm unless --manager npm is selected", + }) + if selected[ManagerNPM] { + if npmLock != "" { + projects = append(projects, Project{Manager: ManagerNPM, Dir: dir, File: npmLock, Name: filepath.Base(dir)}) + } else { + projects = append(projects, Project{Manager: ManagerNPM, Dir: dir, File: shrinkwrap, Name: filepath.Base(dir)}) + } + } + } + } else if npmLock != "" { + projects = append(projects, Project{Manager: ManagerNPM, Dir: dir, File: npmLock, Name: filepath.Base(dir)}) + } else if shrinkwrap != "" { + projects = append(projects, Project{Manager: ManagerNPM, Dir: dir, File: shrinkwrap, Name: filepath.Base(dir)}) + } else if path := files["package.json"]; path != "" && (len(selected) == 0 || selected[ManagerNPM]) { + projects = append(projects, Project{Manager: ManagerNPM, Dir: dir, File: path, Name: filepath.Base(dir)}) + } + } + + sort.Slice(projects, func(i, j int) bool { + if projects[i].Dir != projects[j].Dir { + return projects[i].Dir < projects[j].Dir + } + return projects[i].Manager < projects[j].Manager + }) + if len(projects) == 0 { + return nil, warnings, fmt.Errorf("no supported dependency manifests found under %s", absRoot) + } + return projects, warnings, nil +} + +func discoverManifestFiles(root string, useGit bool) ([]string, error) { + if useGit { + if files, ok := gitManifestFiles(root); ok { + return files, nil + } + } + return walkManifestFiles(root) +} + +func gitManifestFiles(root string) ([]string, bool) { + cmd := osexec.Command("git", "-C", root, "ls-files", "--cached", "--others", "--exclude-standard", "-z", "--", ".") + out, err := cmd.Output() + if err != nil { + return nil, false + } + var files []string + for _, rel := range strings.Split(string(out), "\x00") { + if rel == "" { + continue + } + name := filepath.Base(filepath.FromSlash(rel)) + if managerForManifest(name) == "" { + continue + } + files = append(files, filepath.Join(root, filepath.FromSlash(rel))) + } + sort.Strings(files) + return files, true +} + +func walkManifestFiles(root string) ([]string, error) { + var files []string + err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() { + if path != root && ignoredDirs[d.Name()] { + return filepath.SkipDir + } + return nil + } + name := d.Name() + manager := managerForManifest(name) + if manager == "" { + return nil + } + files = append(files, path) + return nil + }) + if err != nil { + return nil, err + } + sort.Strings(files) + return files, nil +} + +func managerForManifest(name string) Manager { + switch strings.ToLower(name) { + case "go.mod", "go.work": + return ManagerGo + case "pom.xml": + return ManagerMaven + case "build.gradle", "build.gradle.kts": + return ManagerGradle + case "package.json", "package-lock.json", "npm-shrinkwrap.json": + return ManagerNPM + case "pnpm-lock.yaml": + return ManagerPNPM + default: + return "" + } +} + +func managerSet(managers []Manager) map[Manager]bool { + if len(managers) == 0 { + return nil + } + selected := make(map[Manager]bool, len(managers)) + for _, m := range managers { + if m != "" { + selected[m] = true + } + } + return selected +} diff --git a/deps/dockerfile.go b/deps/dockerfile.go new file mode 100644 index 0000000..74fab73 --- /dev/null +++ b/deps/dockerfile.go @@ -0,0 +1,149 @@ +package deps + +import ( + "fmt" + "regexp" + "strings" +) + +// imageRef is a parsed container image reference split into name, tag, and digest. +type imageRef struct { + Name string + Version string + Digest string +} + +var argRefPattern = regexp.MustCompile(`\$\{(\w+)\}|\$(\w+)`) + +// parseDockerfileFrom extracts the external base images referenced by FROM +// directives. It substitutes ARG/ENV defaults declared earlier in the file, +// excludes references to internal build stages (FROM ... AS ) and the +// scratch terminal, and de-duplicates. Unresolved ARG references are warned. +func parseDockerfileFrom(content string) (bases []imageRef, warnings []string) { + args := map[string]string{} + stages := map[string]bool{} + seen := map[string]bool{} + + for _, line := range dockerfileLines(content) { + if line == "" || strings.HasPrefix(line, "#") { + continue + } + fields := strings.Fields(line) + switch strings.ToUpper(fields[0]) { + case "ARG", "ENV": + if name, val, ok := parseArgAssign(fields[1:]); ok { + args[name] = val + } + case "FROM": + ref, stage, ok, warn := parseFromDirective(fields[1:], args, stages) + if warn != "" { + warnings = append(warnings, warn) + } + if stage != "" { + stages[strings.ToLower(stage)] = true + } + if !ok { + continue + } + key := ref.Name + ":" + ref.Version + "@" + ref.Digest + if !seen[key] { + seen[key] = true + bases = append(bases, ref) + } + } + } + return bases, warnings +} + +// parseFromDirective interprets the tokens after FROM: it strips flags +// (--platform=...), resolves the image token, and records an AS stage name. +// ok is false for internal stage references, scratch, and unresolved ARGs. +func parseFromDirective(tokens []string, args map[string]string, stages map[string]bool) (ref imageRef, stage string, ok bool, warn string) { + var image string + for i := 0; i < len(tokens); i++ { + tok := tokens[i] + switch { + case strings.HasPrefix(tok, "--"): + continue + case image == "": + image = tok + case strings.EqualFold(tok, "AS") && i+1 < len(tokens): + stage = tokens[i+1] + i++ + } + } + if image == "" { + return imageRef{}, stage, false, "" + } + resolved, complete := substituteArgs(image, args) + if !complete { + return imageRef{}, stage, false, fmt.Sprintf("unresolved build arg in FROM %q", image) + } + if strings.EqualFold(resolved, "scratch") || stages[strings.ToLower(resolved)] { + return imageRef{}, stage, false, "" + } + return parseImageRef(resolved), stage, true, "" +} + +// substituteArgs replaces ${VAR}/$VAR with known ARG/ENV defaults; complete is +// false when any reference cannot be resolved. +func substituteArgs(s string, args map[string]string) (out string, complete bool) { + complete = true + out = argRefPattern.ReplaceAllStringFunc(s, func(m string) string { + name := strings.Trim(m, "${}") + if v, ok := args[name]; ok { + return v + } + complete = false + return m + }) + return out, complete +} + +// parseImageRef splits an image reference into name, tag, and digest, leaving a +// registry host:port intact. +func parseImageRef(s string) imageRef { + var digest string + if at := strings.Index(s, "@"); at >= 0 { + digest = s[at+1:] + s = s[:at] + } + name, version := s, "" + if i := imageTagSeparator(s); i >= 0 { + name, version = s[:i], s[i+1:] + } + return imageRef{Name: name, Version: version, Digest: digest} +} + +func parseArgAssign(tokens []string) (name, value string, ok bool) { + if len(tokens) == 0 { + return "", "", false + } + eq := strings.SplitN(tokens[0], "=", 2) + if len(eq) != 2 { + return "", "", false + } + return eq[0], strings.Trim(eq[1], `"'`), true +} + +// dockerfileLines joins backslash line continuations and trims carriage returns. +func dockerfileLines(content string) []string { + var lines []string + var buf strings.Builder + for _, ln := range strings.Split(content, "\n") { + ln = strings.TrimRight(ln, "\r") + trimmedRight := strings.TrimRight(ln, " \t") + if strings.HasSuffix(trimmedRight, "\\") { + buf.WriteString(strings.TrimSuffix(trimmedRight, "\\")) + buf.WriteString(" ") + continue + } + buf.WriteString(ln) + lines = append(lines, strings.TrimSpace(buf.String())) + buf.Reset() + } + if buf.Len() > 0 { + lines = append(lines, strings.TrimSpace(buf.String())) + } + return lines +} diff --git a/deps/dockerfile_test.go b/deps/dockerfile_test.go new file mode 100644 index 0000000..de6b770 --- /dev/null +++ b/deps/dockerfile_test.go @@ -0,0 +1,121 @@ +package deps + +import "testing" + +func refsToStrings(refs []imageRef) []string { + out := make([]string, 0, len(refs)) + for _, r := range refs { + s := r.Name + if r.Version != "" { + s += ":" + r.Version + } + if r.Digest != "" { + s += "@" + r.Digest + } + out = append(out, s) + } + return out +} + +func TestParseDockerfileFrom(t *testing.T) { + cases := []struct { + name string + content string + want []string + }{ + { + name: "single FROM with tag", + content: "FROM nginx:1.25.3\nRUN echo hi\n", + want: []string{"nginx:1.25.3"}, + }, + { + name: "bare image no tag", + content: "FROM alpine\n", + want: []string{"alpine"}, + }, + { + name: "multi-stage excludes internal stage reference", + content: "FROM golang:1.22 AS build\n" + + "RUN go build\n" + + "FROM gcr.io/distroless/static:nonroot\n" + + "COPY --from=build /app /app\n", + want: []string{"golang:1.22", "gcr.io/distroless/static:nonroot"}, + }, + { + name: "final FROM referencing a prior stage is internal", + content: "FROM golang:1.22 AS build\n" + + "FROM build\n", + want: []string{"golang:1.22"}, + }, + { + name: "platform flag is stripped", + content: "FROM --platform=linux/amd64 ubuntu:22.04\n", + want: []string{"ubuntu:22.04"}, + }, + { + name: "ARG substitution in tag", + content: "ARG GO_VERSION=1.22\nFROM golang:${GO_VERSION}\n", + want: []string{"golang:1.22"}, + }, + { + name: "ARG substitution in registry and unbraced var", + content: "ARG REG=docker.io\nFROM $REG/library/busybox:1.36\n", + want: []string{"docker.io/library/busybox:1.36"}, + }, + { + name: "scratch is terminal and excluded", + content: "FROM scratch\nCOPY x /\n", + want: nil, + }, + { + name: "digest pinned", + content: "FROM nginx@sha256:abc123\n", + want: []string{"nginx@sha256:abc123"}, + }, + { + name: "tag and digest", + content: "FROM nginx:1.25@sha256:abc123\n", + want: []string{"nginx:1.25@sha256:abc123"}, + }, + { + name: "case-insensitive directives and comments", + content: "# base\nfrom Ubuntu:22.04 as Base\n", + want: []string{"Ubuntu:22.04"}, + }, + { + name: "registry with port keeps host colon", + content: "FROM localhost:5000/team/app:2.0\n", + want: []string{"localhost:5000/team/app:2.0"}, + }, + { + name: "duplicate external bases collapse", + content: "FROM alpine:3.20 AS a\nFROM alpine:3.20 AS b\n", + want: []string{"alpine:3.20"}, + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, _ := parseDockerfileFrom(tc.content) + gotStrings := refsToStrings(got) + if len(gotStrings) != len(tc.want) { + t.Fatalf("bases = %v, want %v", gotStrings, tc.want) + } + for i := range tc.want { + if gotStrings[i] != tc.want[i] { + t.Fatalf("base[%d] = %q, want %q (all: %v)", i, gotStrings[i], tc.want[i], gotStrings) + } + } + }) + } +} + +func TestParseDockerfileUnresolvedArgWarns(t *testing.T) { + bases, warnings := parseDockerfileFrom("FROM $UNDEFINED_BASE\n") + if len(bases) != 0 { + t.Fatalf("unresolved ARG FROM should yield no base, got %v", refsToStrings(bases)) + } + if len(warnings) == 0 { + t.Fatalf("expected a warning for the unresolved ARG reference") + } +} diff --git a/deps/edgegraph.go b/deps/edgegraph.go new file mode 100644 index 0000000..5699c7e --- /dev/null +++ b/deps/edgegraph.go @@ -0,0 +1,89 @@ +package deps + +// edgeGraph is a manager-agnostic adjacency representation of a resolved +// dependency graph. Keys identify nodes (typically Node.ID); Nodes holds a +// childless template per key and Edges lists each node's direct children in a +// stable order. Transitive resolvers (go mod graph, maven dependency:tree, +// gradle dependencies) build an edgeGraph and hand it to buildTreeFromEdgeGraph. +type edgeGraph struct { + RootKey string + Nodes map[string]*Node + Edges map[string][]string +} + +type edgeTreeOptions struct { + Graph edgeGraph + MaxDepth int +} + +// buildTreeFromEdgeGraph expands an edgeGraph into a Node tree. Each node is +// expanded (its children materialized) only at its shallowest BFS occurrence; +// deeper re-occurrences become childless leaves so existing analyzeDuplicates +// can tag them and filterAndPruneAt can mark ancestor-path repeats Circular. +// Children beyond MaxDepth (>0) are pruned during construction. +func buildTreeFromEdgeGraph(opts edgeTreeOptions) *Node { + graph := opts.Graph + if graph.RootKey == "" || graph.Nodes[graph.RootKey] == nil { + return nil + } + depths := edgeGraphDepths(graph) + expanded := map[string]bool{} + + var build func(key string, depth int) *Node + build = func(key string, depth int) *Node { + template := graph.Nodes[key] + if template == nil { + return nil + } + node := template.cloneShallow() + node.Depth = depth + if depth == depths[key] && !expanded[key] { + expanded[key] = true + for _, childKey := range graph.Edges[key] { + if opts.MaxDepth > 0 && depth+1 > opts.MaxDepth { + break + } + if child := build(childKey, depth+1); child != nil { + node.Children = append(node.Children, child) + } + } + } + return node + } + + root := build(graph.RootKey, 0) + sortChildren(root) + return root +} + +// markDirectByDepth flags depth-1 nodes as direct dependencies, matching the +// offline manifest resolvers that mark declared dependencies as direct. +func markDirectByDepth(root *Node) { + if root == nil { + return + } + for _, child := range root.Children { + if child != nil { + child.Direct = child.Depth == 1 + } + } +} + +// edgeGraphDepths returns the shortest-path depth of every reachable node from +// RootKey via breadth-first traversal. +func edgeGraphDepths(graph edgeGraph) map[string]int { + depths := map[string]int{graph.RootKey: 0} + queue := []string{graph.RootKey} + for len(queue) > 0 { + key := queue[0] + queue = queue[1:] + for _, child := range graph.Edges[key] { + if _, seen := depths[child]; seen { + continue + } + depths[child] = depths[key] + 1 + queue = append(queue, child) + } + } + return depths +} diff --git a/deps/edgegraph_test.go b/deps/edgegraph_test.go new file mode 100644 index 0000000..a608f84 --- /dev/null +++ b/deps/edgegraph_test.go @@ -0,0 +1,99 @@ +package deps + +import "testing" + +// diamondGraph: root -> a, b; a -> shared; b -> shared. +func diamondGraph() edgeGraph { + mk := func(name string) *Node { return NewNode(ManagerGo, name, "v1") } + keyA := mk("a").ID + keyB := mk("b").ID + keyShared := mk("shared").ID + root := mk("root") + return edgeGraph{ + RootKey: root.ID, + Nodes: map[string]*Node{ + root.ID: root, + keyA: mk("a"), + keyB: mk("b"), + keyShared: mk("shared"), + }, + Edges: map[string][]string{ + root.ID: {keyA, keyB}, + keyA: {keyShared}, + keyB: {keyShared}, + }, + } +} + +func TestEdgeGraphDepthsDiamond(t *testing.T) { + graph := diamondGraph() + depths := edgeGraphDepths(graph) + if depths[NewNode(ManagerGo, "shared", "v1").ID] != 2 { + t.Fatalf("shared depth = %d, want 2", depths[NewNode(ManagerGo, "shared", "v1").ID]) + } + if depths[NewNode(ManagerGo, "a", "v1").ID] != 1 { + t.Fatalf("a depth = %d, want 1", depths[NewNode(ManagerGo, "a", "v1").ID]) + } +} + +func TestBuildTreeFirstOccurrenceExpandsOnceDuplicateLeaf(t *testing.T) { + root := buildTreeFromEdgeGraph(edgeTreeOptions{Graph: diamondGraph()}) + if root == nil || len(root.Children) != 2 { + t.Fatalf("root children = %#v", root) + } + a := findChild(root, "a") + b := findChild(root, "b") + occurrences := 0 + for _, branch := range []*Node{a, b} { + shared := findChild(branch, "shared") + if shared == nil { + continue + } + occurrences++ + if len(shared.Children) != 0 { + t.Fatalf("shared should be a childless leaf at re-occurrence, got %#v", shared) + } + } + if occurrences != 2 { + t.Fatalf("shared should appear under both branches, got %d", occurrences) + } +} + +func TestBuildTreeTerminatesOnCycle(t *testing.T) { + mk := func(name string) *Node { return NewNode(ManagerGo, name, "v1") } + rootKey := mk("root").ID + aKey := mk("a").ID + graph := edgeGraph{ + RootKey: rootKey, + Nodes: map[string]*Node{rootKey: mk("root"), aKey: mk("a")}, + Edges: map[string][]string{ + rootKey: {aKey}, + aKey: {rootKey}, // cycle a -> root + }, + } + root := buildTreeFromEdgeGraph(edgeTreeOptions{Graph: graph}) + a := findChild(root, "a") + if a == nil { + t.Fatalf("expected child a, got %#v", root) + } + // root re-occurs as a leaf under a (deeper than its BFS depth 0), not expanded. + rootLeaf := findChild(a, "root") + if rootLeaf == nil || len(rootLeaf.Children) != 0 { + t.Fatalf("cycle back-edge should be a leaf, got %#v", rootLeaf) + } + // filterAndPrune marks the ancestor-path repeat as circular. + pruned := filterAndPrune(root, nil, 0) + if leaf := findChild(findChild(pruned, "a"), "root"); leaf == nil || !leaf.Circular { + t.Fatalf("expected circular marker on back-edge, got %#v", leaf) + } +} + +func TestBuildTreeMaxDepthPruning(t *testing.T) { + root := buildTreeFromEdgeGraph(edgeTreeOptions{Graph: diamondGraph(), MaxDepth: 1}) + if root == nil || len(root.Children) != 2 { + t.Fatalf("root children = %#v", root) + } + if a := findChild(root, "a"); a == nil || len(a.Children) != 0 { + t.Fatalf("depth-1 child should have no children at MaxDepth 1, got %#v", a) + } +} diff --git a/deps/filter.go b/deps/filter.go new file mode 100644 index 0000000..ddb3216 --- /dev/null +++ b/deps/filter.go @@ -0,0 +1,237 @@ +package deps + +import ( + "fmt" + "sort" + + "github.com/flanksource/commons/collections" +) + +func filterAndPrune(root *Node, filters []string, maxDepth int) *Node { + return filterAndPruneAt(root, filters, maxDepth, nil) +} + +func filterAndPruneAt(node *Node, filters []string, maxDepth int, path map[string]bool) *Node { + if node == nil { + return nil + } + if maxDepth > 0 && node.Depth > maxDepth { + return nil + } + if path == nil { + path = map[string]bool{} + } + cp := node.cloneShallow() + key := node.ID + if path[key] { + cp.Circular = true + return cp + } + path[key] = true + for _, child := range node.Children { + if filtered := filterAndPruneAt(child, filters, maxDepth, cloneBoolMap(path)); filtered != nil { + cp.Children = append(cp.Children, filtered) + } + } + if len(filters) == 0 || nodeMatches(cp, filters) || len(cp.Children) > 0 || cp.Depth == 0 { + return cp + } + return nil +} + +func nodeMatches(node *Node, filters []string) bool { + if len(filters) == 0 { + return true + } + candidates := []string{ + node.ID, + node.Name, + node.Version, + string(node.Manager), + node.Scope, + node.Source, + node.Path, + fmt.Sprintf("%s:%s", node.Manager, node.Name), + fmt.Sprintf("%s:%s@%s", node.Manager, node.Name, node.Version), + } + for _, candidate := range candidates { + if candidate != "" && collections.MatchItems(candidate, filters...) { + return true + } + } + return false +} + +func flatten(roots []*Node, duplicates map[string]*Duplicate) ([]FlatNode, []Edge, Statistics) { + nodeMap := map[string]FlatNode{} + edgeMap := map[string]Edge{} + stats := Statistics{ByManager: map[Manager]int{}} + var circular int + var visit func(*Node) + visit = func(node *Node) { + if node == nil { + return + } + if _, exists := nodeMap[node.ID]; !exists { + nodeMap[node.ID] = FlatNode{ + ID: node.ID, + Name: node.Name, + Version: node.Version, + Manager: node.Manager, + Scope: node.Scope, + Source: node.Source, + Path: node.Path, + Direct: node.Direct, + Dev: node.Dev, + Optional: node.Optional, + Local: node.Local, + Depth: node.Depth, + } + stats.ByManager[node.Manager]++ + if node.Depth > stats.MaxDepth { + stats.MaxDepth = node.Depth + } + if node.Circular { + circular++ + } + } + for _, child := range node.Children { + edgeKey := node.ID + ">" + child.ID + ">" + child.Scope + edgeMap[edgeKey] = Edge{ + From: node.ID, + To: child.ID, + Manager: child.Manager, + Scope: child.Scope, + Dev: child.Dev, + Optional: child.Optional, + } + visit(child) + } + } + for _, root := range roots { + visit(root) + } + nodes := make([]FlatNode, 0, len(nodeMap)) + for _, node := range nodeMap { + nodes = append(nodes, node) + } + sort.Slice(nodes, func(i, j int) bool { + if nodes[i].Manager != nodes[j].Manager { + return nodes[i].Manager < nodes[j].Manager + } + if nodes[i].Depth != nodes[j].Depth { + return nodes[i].Depth < nodes[j].Depth + } + return nodes[i].ID < nodes[j].ID + }) + edges := make([]Edge, 0, len(edgeMap)) + for _, edge := range edgeMap { + edges = append(edges, edge) + } + sort.Slice(edges, func(i, j int) bool { + if edges[i].From != edges[j].From { + return edges[i].From < edges[j].From + } + if edges[i].To != edges[j].To { + return edges[i].To < edges[j].To + } + return edges[i].Scope < edges[j].Scope + }) + stats.Total = len(nodes) + stats.Edges = len(edges) + stats.Circular = circular + for _, dup := range duplicates { + if dup.Count > 1 { + stats.Duplicates++ + if dup.Conflicts { + stats.Conflicts++ + } + } + } + return nodes, edges, stats +} + +func analyzeDuplicates(roots []*Node) map[string]*Duplicate { + out := map[string]*Duplicate{} + var walk func(*Node, string) + walk = func(node *Node, parentPath string) { + if node == nil { + return + } + currentPath := node.Name + if parentPath != "" { + currentPath = parentPath + " > " + node.Name + } + key := string(node.Manager) + ":" + node.Name + dup := out[key] + if dup == nil { + dup = &Duplicate{ + Name: node.Name, + Manager: node.Manager, + Versions: map[string][]string{}, + } + out[key] = dup + } + dup.Count++ + version := node.Version + if version == "" { + version = "(none)" + } + dup.Versions[version] = append(dup.Versions[version], currentPath) + for _, child := range node.Children { + walk(child, currentPath) + } + } + for _, root := range roots { + walk(root, "") + } + for _, dup := range out { + if len(dup.Versions) > 1 { + dup.Conflicts = true + } + } + return out +} + +func applyDuplicateRefs(roots []*Node, duplicates map[string]*Duplicate) { + var walk func(*Node) + walk = func(node *Node) { + if node == nil { + return + } + key := string(node.Manager) + ":" + node.Name + if dup := duplicates[key]; dup != nil && dup.Count > 1 { + node.Duplicate = &DupRef{Count: dup.Count, Conflicts: dup.Conflicts} + } + for _, child := range node.Children { + walk(child) + } + } + for _, root := range roots { + walk(root) + } +} + +func duplicatesList(duplicates map[string]*Duplicate) []Duplicate { + out := make([]Duplicate, 0) + for _, dup := range duplicates { + if dup.Count > 1 { + out = append(out, *dup) + } + } + sort.Slice(out, func(i, j int) bool { + if out[i].Manager != out[j].Manager { + return out[i].Manager < out[j].Manager + } + return out[i].Name < out[j].Name + }) + return out +} + +func cloneBoolMap(in map[string]bool) map[string]bool { + out := make(map[string]bool, len(in)) + for k, v := range in { + out[k] = v + } + return out +} diff --git a/deps/go.go b/deps/go.go new file mode 100644 index 0000000..7ca88a9 --- /dev/null +++ b/deps/go.go @@ -0,0 +1,62 @@ +package deps + +import ( + "fmt" + "os" + "path/filepath" + + "golang.org/x/mod/modfile" +) + +func resolveGoManifest(project Project, opts Options) (*Node, []Warning, error) { + file, err := loadGoModFile(project.Dir) + if err != nil { + return nil, nil, err + } + root := NewNode(ManagerGo, file.Module.Mod.Path, "") + root.Path = filepath.Join(project.Dir, "go.mod") + root.Source = "go.mod" + for _, req := range file.Require { + if req.Indirect && !opts.IncludeIndirect { + continue + } + child := NewNode(ManagerGo, req.Mod.Path, req.Mod.Version) + child.Depth = 1 + child.Direct = !req.Indirect + child.Scope = "require" + if req.Indirect { + child.Scope = "indirect" + } + if rep := goReplaceFor(file, req.Mod.Path, req.Mod.Version); rep != nil { + child.Source = goReplaceSource(rep.New.Path, rep.New.Version) + child.Local = isLocalRef(rep.New.Path) + } + root.Children = append(root.Children, child) + } + sortChildren(root) + return root, []Warning{{Manager: ManagerGo, Project: project.Dir, Message: "offline go.mod parsing includes declared requirements only; transitive edges are unavailable"}}, nil +} + +func loadGoModFile(dir string) (*modfile.File, error) { + data, err := os.ReadFile(filepath.Join(dir, "go.mod")) + if err != nil { + return nil, err + } + return modfile.Parse("go.mod", data, nil) +} + +func goReplaceFor(file *modfile.File, path, version string) *modfile.Replace { + for _, rep := range file.Replace { + if rep.Old.Path == path && (rep.Old.Version == "" || rep.Old.Version == version) { + return rep + } + } + return nil +} + +func goReplaceSource(path, version string) string { + if version == "" { + return path + } + return fmt.Sprintf("%s@%s", path, version) +} diff --git a/deps/go_graph.go b/deps/go_graph.go new file mode 100644 index 0000000..70bd74b --- /dev/null +++ b/deps/go_graph.go @@ -0,0 +1,134 @@ +package deps + +import ( + "context" + "fmt" + "os/exec" + "path/filepath" + "strings" + + "golang.org/x/mod/modfile" +) + +// resolveGoGraph resolves the transitive Go dependency graph by shelling out to +// `go mod graph`. It fails fast with a toolError when the go binary is missing +// or the command fails, suggesting --depth 1 for offline direct-only output. +func resolveGoGraph(ctx context.Context, project Project, opts Options) (*Node, []Warning, error) { + if _, err := exec.LookPath("go"); err != nil { + return nil, nil, toolError{fmt.Errorf("go binary not found on PATH; rerun with --depth 1 for offline direct-only output: %w", err)} + } + file, err := loadGoModFile(project.Dir) + if err != nil { + return nil, nil, err + } + mainModule := file.Module.Mod.Path + + result, err := opts.Runner.Run(ctx, Command{ + Dir: project.Dir, + Name: "go", + Args: []string{"mod", "graph"}, + Env: []string{"GOFLAGS=-mod=mod"}, + }) + if err != nil { + detail := strings.TrimSpace(result.Stderr) + if detail == "" { + detail = err.Error() + } + return nil, nil, toolError{fmt.Errorf("go mod graph failed in %s (rerun with --depth 1 for offline direct-only output): %s", project.Dir, detail)} + } + + graph, err := parseGoModGraph(result.Stdout, mainModule) + if err != nil { + return nil, nil, toolError{fmt.Errorf("go mod graph parse failed in %s: %w", project.Dir, err)} + } + + root := buildTreeFromEdgeGraph(edgeTreeOptions{Graph: graph, MaxDepth: opts.MaxDepth}) + if root == nil { + return nil, nil, toolError{fmt.Errorf("go mod graph produced no nodes for %s", mainModule)} + } + root.Path = filepath.Join(project.Dir, "go.mod") + root.Source = "go.mod" + applyGoModMetadata(root, file) + + warning := Warning{ + Manager: ManagerGo, + Project: project.Dir, + Message: "go mod graph reports the module requirement graph (MVS inputs); it may list module versions that are not selected in the final build list", + } + return root, []Warning{warning}, nil +} + +// parseGoModGraph converts `go mod graph` output into an edgeGraph. Each line is +// " " where a node is "path@version" (the main module appears without +// a version). The root key is the main module. +func parseGoModGraph(output, mainModule string) (edgeGraph, error) { + graph := edgeGraph{ + RootKey: NodeID(ManagerGo, mainModule, ""), + Nodes: map[string]*Node{}, + Edges: map[string][]string{}, + } + graph.Nodes[graph.RootKey] = NewNode(ManagerGo, mainModule, "") + + seenEdge := map[string]bool{} + for _, line := range strings.Split(output, "\n") { + line = strings.TrimSpace(line) + if line == "" { + continue + } + fields := strings.Fields(line) + if len(fields) != 2 { + return edgeGraph{}, fmt.Errorf("malformed go mod graph line: %q", line) + } + fromKey := goModNodeKey(&graph, fields[0]) + toKey := goModNodeKey(&graph, fields[1]) + edgeKey := fromKey + ">" + toKey + if seenEdge[edgeKey] { + continue + } + seenEdge[edgeKey] = true + graph.Edges[fromKey] = append(graph.Edges[fromKey], toKey) + } + return graph, nil +} + +func goModNodeKey(graph *edgeGraph, field string) string { + path, version, _ := strings.Cut(field, "@") + key := NodeID(ManagerGo, path, version) + if _, ok := graph.Nodes[key]; !ok { + graph.Nodes[key] = NewNode(ManagerGo, path, version) + } + return key +} + +// applyGoModMetadata overlays go.mod metadata onto the resolved graph: direct vs +// indirect requirement scope and replace directives (Source/Local). +func applyGoModMetadata(root *Node, file *modfile.File) { + requires := map[string]*modfile.Require{} + for _, req := range file.Require { + requires[req.Mod.Path] = req + } + var walk func(n *Node) + walk = func(n *Node) { + if n == nil { + return + } + if req, ok := requires[n.Name]; ok { + n.Direct = !req.Indirect + if req.Indirect { + n.Scope = "indirect" + } else { + n.Scope = "require" + } + } + if rep := goReplaceFor(file, n.Name, n.Version); rep != nil { + n.Source = goReplaceSource(rep.New.Path, rep.New.Version) + n.Local = isLocalRef(rep.New.Path) + } + for _, child := range n.Children { + walk(child) + } + } + for _, child := range root.Children { + walk(child) + } +} diff --git a/deps/go_graph_test.go b/deps/go_graph_test.go new file mode 100644 index 0000000..4bf9948 --- /dev/null +++ b/deps/go_graph_test.go @@ -0,0 +1,232 @@ +package deps + +import ( + "context" + "fmt" + "path/filepath" + "strings" + "testing" + "time" +) + +type fakeRunner struct { + result CommandResult + err error + calls []Command + onRun func(Command) (CommandResult, error) +} + +func (f *fakeRunner) Run(_ context.Context, cmd Command) (CommandResult, error) { + f.calls = append(f.calls, cmd) + if f.onRun != nil { + return f.onRun(cmd) + } + return f.result, f.err +} + +func TestParseGoModGraph(t *testing.T) { + output := strings.Join([]string{ + "github.com/acme/app github.com/acme/lib@v1.2.3", + "github.com/acme/app github.com/acme/other@v0.5.0", + "github.com/acme/lib@v1.2.3 github.com/acme/dep@v0.1.0", + "github.com/acme/other@v0.5.0 github.com/acme/dep@v0.1.0", + "", + }, "\n") + + graph, err := parseGoModGraph(output, "github.com/acme/app") + if err != nil { + t.Fatal(err) + } + rootKey := NodeID(ManagerGo, "github.com/acme/app", "") + if graph.RootKey != rootKey { + t.Fatalf("root key = %q, want %q", graph.RootKey, rootKey) + } + if len(graph.Edges[rootKey]) != 2 { + t.Fatalf("root edges = %d, want 2", len(graph.Edges[rootKey])) + } + depKey := NodeID(ManagerGo, "github.com/acme/dep", "v0.1.0") + depths := edgeGraphDepths(graph) + if depths[depKey] != 2 { + t.Fatalf("dep depth = %d, want 2 (diamond shortest path)", depths[depKey]) + } +} + +func TestResolveGoGraphMergesGoModMetadata(t *testing.T) { + dir := t.TempDir() + writeFile(t, filepath.Join(dir, "go.mod"), `module github.com/acme/app + +go 1.22 + +require ( + github.com/acme/lib v1.2.3 + github.com/acme/indirect v0.1.0 // indirect +) + +replace github.com/acme/lib => ../lib +`) + runner := &fakeRunner{result: CommandResult{Stdout: strings.Join([]string{ + "github.com/acme/app github.com/acme/lib@v1.2.3", + "github.com/acme/app github.com/acme/indirect@v0.1.0", + "github.com/acme/lib@v1.2.3 github.com/acme/dep@v0.3.0", + "", + }, "\n")}} + + root, warnings, err := resolveGoGraph(context.Background(), Project{Manager: ManagerGo, Dir: dir}, Options{Runner: runner, MaxDepth: 0}) + if err != nil { + t.Fatal(err) + } + if len(warnings) != 1 || !strings.Contains(warnings[0].Message, "MVS") { + t.Fatalf("expected MVS warning, got %#v", warnings) + } + lib := findChild(root, "github.com/acme/lib") + if lib == nil || !lib.Direct || lib.Scope != "require" || !lib.Local || lib.Source != "../lib" { + t.Fatalf("lib metadata not merged: %#v", lib) + } + indirect := findChild(root, "github.com/acme/indirect") + if indirect == nil || indirect.Direct || indirect.Scope != "indirect" { + t.Fatalf("indirect metadata not merged: %#v", indirect) + } + dep := findChild(lib, "github.com/acme/dep") + if dep == nil || dep.Depth != 2 { + t.Fatalf("transitive dep not resolved at depth 2: %#v", dep) + } +} + +func TestResolveGoGraphToolFailureIsFatal(t *testing.T) { + dir := t.TempDir() + writeFile(t, filepath.Join(dir, "go.mod"), `module github.com/acme/app + +go 1.22 + +require github.com/acme/lib v1.2.3 +`) + runner := &fakeRunner{err: fmt.Errorf("exit status 1"), result: CommandResult{Stderr: "go: updates to go.mod needed"}} + + _, err := Scan(context.Background(), dir, Options{ + Managers: []Manager{ManagerGo}, + MaxDepth: 0, + Runner: runner, + Now: func() time.Time { return time.Unix(1, 0).UTC() }, + }) + if err == nil { + t.Fatal("expected fatal tool error to abort scan") + } + if !strings.Contains(err.Error(), "--depth 1") { + t.Fatalf("error should suggest --depth 1, got %v", err) + } +} + +// goDiamondScan resolves a go module whose two direct dependencies both require +// the same transitive dep, exercising the collapse pass end-to-end. +func goDiamondScan(t *testing.T, opts Options) *Export { + t.Helper() + dir := t.TempDir() + writeFile(t, filepath.Join(dir, "go.mod"), `module github.com/acme/app + +go 1.22 + +require ( + github.com/acme/lib v1.0.0 + github.com/acme/other v1.0.0 +) +`) + opts.Managers = []Manager{ManagerGo} + opts.MaxDepth = 0 + opts.Runner = &fakeRunner{result: CommandResult{Stdout: strings.Join([]string{ + "github.com/acme/app github.com/acme/lib@v1.0.0", + "github.com/acme/app github.com/acme/other@v1.0.0", + "github.com/acme/lib@v1.0.0 github.com/acme/dep@v0.1.0", + "github.com/acme/other@v1.0.0 github.com/acme/dep@v0.1.0", + "", + }, "\n")}} + opts.Now = func() time.Time { return time.Unix(1, 0).UTC() } + got, err := Scan(context.Background(), dir, opts) + if err != nil { + t.Fatal(err) + } + return got +} + +func TestScanCollapsesDuplicatesSilentlyByDefault(t *testing.T) { + got := goDiamondScan(t, Options{}) + root := got.Roots[0] + lib := findChild(root, "github.com/acme/lib") + other := findChild(root, "github.com/acme/other") + + depUnderLib := findChild(lib, "github.com/acme/dep") + depUnderOther := findChild(other, "github.com/acme/dep") + if (depUnderLib == nil) == (depUnderOther == nil) { + t.Fatalf("dep should render under exactly one parent, lib=%v other=%v", depUnderLib != nil, depUnderOther != nil) + } + if got.Statistics.Duplicates != 0 { + t.Fatalf("default scan should not count duplicates, got %d", got.Statistics.Duplicates) + } + if len(got.Duplicates) != 0 { + t.Fatalf("default scan should not report a duplicates section, got %d", len(got.Duplicates)) + } +} + +func TestScanShowDuplicatesRendersEveryOccurrence(t *testing.T) { + got := goDiamondScan(t, Options{ShowDuplicates: true}) + root := got.Roots[0] + lib := findChild(root, "github.com/acme/lib") + other := findChild(root, "github.com/acme/other") + if findChild(lib, "github.com/acme/dep") == nil || findChild(other, "github.com/acme/dep") == nil { + t.Fatalf("--show-duplicates should keep dep under both parents") + } + if !got.Metadata.ShowDuplicates { + t.Fatalf("metadata should record show_duplicates") + } + if got.Statistics.Duplicates == 0 { + t.Fatalf("--show-duplicates should count the shared dep as a duplicate") + } + if len(got.Duplicates) == 0 { + t.Fatalf("--show-duplicates should populate the duplicates section") + } +} + +func TestScanFlatStaysFullGraph(t *testing.T) { + collapsed := goDiamondScan(t, Options{}) + flat := goDiamondScan(t, Options{Flat: true}) + // The flat export counts every node once (dedup by id), independent of the + // tree collapse, so dep is present exactly once and edges cover both parents. + var depEdges int + for _, e := range flat.Edges { + if strings.HasSuffix(e.To, "acme/dep@v0.1.0") { + depEdges++ + } + } + if depEdges != 2 { + t.Fatalf("flat edges should retain both parent→dep edges, got %d", depEdges) + } + if len(collapsed.Roots) == 0 || len(flat.Nodes) == 0 { + t.Fatalf("expected collapsed roots and flat nodes to be populated") + } +} + +func TestScanGoDepthOneStaysOffline(t *testing.T) { + dir := t.TempDir() + writeFile(t, filepath.Join(dir, "go.mod"), `module github.com/acme/app + +go 1.22 + +require github.com/acme/lib v1.2.3 +`) + runner := &fakeRunner{} + + got, err := Scan(context.Background(), dir, Options{ + Managers: []Manager{ManagerGo}, + MaxDepth: 1, + Runner: runner, + Now: func() time.Time { return time.Unix(1, 0).UTC() }, + }) + if err != nil { + t.Fatal(err) + } + if len(runner.calls) != 0 { + t.Fatalf("runner should not be called at depth 1, got %d calls", len(runner.calls)) + } + if len(got.Roots) != 1 { + t.Fatalf("roots = %d, want 1", len(got.Roots)) + } +} diff --git a/deps/go_test.go b/deps/go_test.go new file mode 100644 index 0000000..3d2b7c8 --- /dev/null +++ b/deps/go_test.go @@ -0,0 +1,49 @@ +package deps + +import ( + "path/filepath" + "testing" +) + +const goModWithIndirect = `module github.com/acme/app + +go 1.22 + +require ( + github.com/acme/lib v1.2.3 + github.com/acme/indirect v0.1.0 // indirect +) +` + +func TestGoManifestExcludesIndirectByDefault(t *testing.T) { + dir := t.TempDir() + writeFile(t, filepath.Join(dir, "go.mod"), goModWithIndirect) + + root, _, err := resolveGoManifest(Project{Manager: ManagerGo, Dir: dir}, Options{}) + if err != nil { + t.Fatal(err) + } + if len(root.Children) != 1 { + t.Fatalf("children = %d, want 1 (indirect excluded)", len(root.Children)) + } + if root.Children[0].Name != "github.com/acme/lib" { + t.Fatalf("unexpected direct child: %#v", root.Children[0]) + } +} + +func TestGoManifestIncludeIndirect(t *testing.T) { + dir := t.TempDir() + writeFile(t, filepath.Join(dir, "go.mod"), goModWithIndirect) + + root, _, err := resolveGoManifest(Project{Manager: ManagerGo, Dir: dir}, Options{IncludeIndirect: true}) + if err != nil { + t.Fatal(err) + } + if len(root.Children) != 2 { + t.Fatalf("children = %d, want 2 (indirect included)", len(root.Children)) + } + indirect := findChild(root, "github.com/acme/indirect") + if indirect == nil || indirect.Direct || indirect.Scope != "indirect" { + t.Fatalf("indirect metadata not captured: %#v", indirect) + } +} diff --git a/deps/gradle.go b/deps/gradle.go new file mode 100644 index 0000000..488420a --- /dev/null +++ b/deps/gradle.go @@ -0,0 +1,41 @@ +package deps + +import ( + "os" + "path/filepath" + "regexp" + "strings" +) + +func resolveGradleManifest(project Project) (*Node, []Warning, error) { + root, err := parseGradleBuildFile(project.File) + if err != nil { + return nil, nil, err + } + return root, []Warning{{Manager: ManagerGradle, Project: project.Dir, Message: "offline Gradle build-file parsing includes direct dependency declarations only; resolved transitive edges are unavailable"}}, nil +} + +var gradleDepRe = regexp.MustCompile(`(?m)^\s*([A-Za-z][A-Za-z0-9_]*(?:Implementation|CompileOnly|RuntimeOnly|Api|TestImplementation|testImplementation|implementation|api|compileOnly|runtimeOnly|testRuntimeOnly|annotationProcessor|kapt)?)\s*(?:\(?\s*)["']([^:"']+):([^:"']+):([^"']+)["']`) + +func parseGradleBuildFile(path string) (*Node, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + root := NewNode(ManagerGradle, filepath.Base(filepath.Dir(path)), "") + root.Path = path + root.Source = filepath.Base(path) + for _, match := range gradleDepRe.FindAllStringSubmatch(string(data), -1) { + scope := match[1] + name := match[2] + ":" + match[3] + version := match[4] + child := NewNode(ManagerGradle, name, version) + child.Depth = 1 + child.Direct = true + child.Scope = scope + child.Dev = strings.Contains(strings.ToLower(scope), "test") + root.Children = append(root.Children, child) + } + sortChildren(root) + return root, nil +} diff --git a/deps/gradle_tree.go b/deps/gradle_tree.go new file mode 100644 index 0000000..7f2a3c5 --- /dev/null +++ b/deps/gradle_tree.go @@ -0,0 +1,202 @@ +package deps + +import ( + "context" + "fmt" + "os" + "path/filepath" + "strings" +) + +const gradleDefaultConfiguration = "runtimeClasspath" + +// resolveGradleTree resolves the transitive Gradle dependency graph by running +// the `dependencies` task for the runtimeClasspath configuration. It fails fast +// with a toolError when gradle is missing, the command fails, or the +// configuration is absent, suggesting --depth 1 for offline output. +func resolveGradleTree(ctx context.Context, project Project, opts Options) (*Node, []Warning, error) { + command := gradleCommand(project.Dir) + result, err := opts.Runner.Run(ctx, Command{ + Dir: project.Dir, + Name: command, + Args: []string{"-q", "dependencies", "--configuration", gradleDefaultConfiguration}, + }) + if err != nil { + detail := strings.TrimSpace(result.Stderr) + if detail == "" { + detail = err.Error() + } + return nil, nil, toolError{fmt.Errorf("gradle dependencies --configuration %s failed in %s (rerun with --depth 1 for offline direct-only output): %s", gradleDefaultConfiguration, project.Dir, detail)} + } + + graph, err := parseGradleDependencyTree(result.Stdout, filepath.Base(filepath.Dir(project.File))) + if err != nil { + return nil, nil, toolError{fmt.Errorf("gradle dependencies parse failed in %s: %w", project.Dir, err)} + } + root := buildTreeFromEdgeGraph(edgeTreeOptions{Graph: graph, MaxDepth: opts.MaxDepth}) + if root == nil { + return nil, nil, toolError{fmt.Errorf("gradle dependencies produced no nodes for %s", project.Dir)} + } + root.Path = project.File + root.Source = filepath.Base(project.File) + applyGradleScope(root) + markDirectByDepth(root) + return root, nil, nil +} + +// gradleCommand prefers a ./gradlew wrapper found by walking up from dir, and +// falls back to a gradle binary on PATH. +func gradleCommand(dir string) string { + current := dir + for { + wrapper := filepath.Join(current, "gradlew") + if info, err := os.Stat(wrapper); err == nil && !info.IsDir() { + return wrapper + } + parent := filepath.Dir(current) + if parent == current { + break + } + current = parent + } + return "gradle" +} + +// parseGradleDependencyTree parses the indented ASCII tree printed by the gradle +// `dependencies` task into an edgeGraph rooted at a synthetic project node. +// It resolves "-> version" overrides, treats "(*)" repeated subtrees as leaves, +// skips "(c)" constraint and "(n)" unresolved entries, and marks "project :x" +// nodes as local. +func parseGradleDependencyTree(output, rootName string) (edgeGraph, error) { + rootKey := NodeID(ManagerGradle, rootName, "") + graph := edgeGraph{ + RootKey: rootKey, + Nodes: map[string]*Node{rootKey: NewNode(ManagerGradle, rootName, "")}, + Edges: map[string][]string{}, + } + type frame struct { + depth int + key string + } + stack := []frame{} + seenEdge := map[string]bool{} + + for _, raw := range strings.Split(output, "\n") { + col, ok := gradleMarkerColumn(raw) + if !ok { + continue + } + depth := col/gradleIndentWidth + 1 + entry := strings.TrimSpace(raw[col+gradleMarkerWidth:]) + node, skip := gradleNodeFromEntry(entry) + if skip { + continue + } + key := node.ID + if _, exists := graph.Nodes[key]; !exists { + graph.Nodes[key] = node + } + for len(stack) > 0 && stack[len(stack)-1].depth >= depth { + stack = stack[:len(stack)-1] + } + parentKey := rootKey + if len(stack) > 0 { + parentKey = stack[len(stack)-1].key + } + edgeKey := parentKey + ">" + key + if !seenEdge[edgeKey] { + seenEdge[edgeKey] = true + graph.Edges[parentKey] = append(graph.Edges[parentKey], key) + } + stack = append(stack, frame{depth: depth, key: key}) + } + return graph, nil +} + +const ( + gradleIndentWidth = 5 + gradleMarkerWidth = 5 // "+--- " or "\--- " +) + +func gradleMarkerColumn(line string) (int, bool) { + for _, marker := range []string{"+--- ", "\\--- "} { + idx := strings.Index(line, marker) + if idx < 0 { + continue + } + if gradleIsIndent(line[:idx]) { + return idx, true + } + } + return 0, false +} + +func gradleIsIndent(prefix string) bool { + for _, r := range prefix { + if r != ' ' && r != '|' { + return false + } + } + return true +} + +func gradleNodeFromEntry(entry string) (*Node, bool) { + entry = strings.TrimSpace(entry) + if entry == "" { + return nil, true + } + if strings.HasSuffix(entry, "(c)") || strings.HasSuffix(entry, "(n)") { + return nil, true + } + entry = strings.TrimSpace(strings.TrimSuffix(entry, "(*)")) + if strings.HasPrefix(entry, "project ") { + name := strings.TrimSpace(strings.TrimPrefix(entry, "project")) + node := NewNode(ManagerGradle, name, "") + node.Local = true + return node, false + } + left, right, hasArrow := strings.Cut(entry, " -> ") + name, version := gradleCoordinate(left) + if hasArrow { + version = gradleResolvedVersion(strings.TrimSpace(right)) + } + return NewNode(ManagerGradle, name, version), false +} + +func gradleCoordinate(text string) (name, version string) { + parts := strings.Split(strings.TrimSpace(text), ":") + switch len(parts) { + case 3: + return parts[0] + ":" + parts[1], parts[2] + case 2: + return parts[0] + ":" + parts[1], "" + default: + return strings.TrimSpace(text), "" + } +} + +func gradleResolvedVersion(right string) string { + if strings.Contains(right, ":") { + parts := strings.Split(right, ":") + return parts[len(parts)-1] + } + return right +} + +func applyGradleScope(root *Node) { + var walk func(n *Node) + walk = func(n *Node) { + if n == nil { + return + } + if n.Scope == "" { + n.Scope = gradleDefaultConfiguration + } + for _, child := range n.Children { + walk(child) + } + } + for _, child := range root.Children { + walk(child) + } +} diff --git a/deps/gradle_tree_test.go b/deps/gradle_tree_test.go new file mode 100644 index 0000000..4c981be --- /dev/null +++ b/deps/gradle_tree_test.go @@ -0,0 +1,84 @@ +package deps + +import ( + "context" + "errors" + "fmt" + "path/filepath" + "strings" + "testing" +) + +const gradleTreeFixture = ` +runtimeClasspath - Runtime classpath of source set 'main'. ++--- org.foo:bar:1.0 +| \--- org.baz:qux:2.0 ++--- org.abc:def:1.0 -> 1.2 +\--- project :submodule + \--- org.x:y:3.0 (*) + +(*) - Indicates repeated occurrences of a transitive dependency subtree. +` + +func TestParseGradleDependencyTree(t *testing.T) { + graph, err := parseGradleDependencyTree(gradleTreeFixture, "app") + if err != nil { + t.Fatal(err) + } + root := buildTreeFromEdgeGraph(edgeTreeOptions{Graph: graph}) + applyGradleScope(root) + + bar := findChild(root, "org.foo:bar") + if bar == nil || bar.Version != "1.0" { + t.Fatalf("bar not parsed: %#v", bar) + } + qux := findChild(bar, "org.baz:qux") + if qux == nil || qux.Depth != 2 { + t.Fatalf("transitive qux not at depth 2: %#v", qux) + } + def := findChild(root, "org.abc:def") + if def == nil || def.Version != "1.2" { + t.Fatalf("resolved-version override not applied: %#v", def) + } + sub := findChild(root, ":submodule") + if sub == nil || !sub.Local { + t.Fatalf("project dependency not marked local: %#v", sub) + } + if root.Children[0].Scope != gradleDefaultConfiguration { + t.Fatalf("scope not applied: %#v", root.Children[0]) + } +} + +func TestGradleCommandPrefersWrapper(t *testing.T) { + root := t.TempDir() + writeFile(t, filepath.Join(root, "gradlew"), "#!/bin/sh\n") + sub := filepath.Join(root, "service") + writeFile(t, filepath.Join(sub, "build.gradle"), "") + + got := gradleCommand(sub) + if got != filepath.Join(root, "gradlew") { + t.Fatalf("gradleCommand = %q, want wrapper at repo root", got) + } + + bare := t.TempDir() + if gradleCommand(bare) != "gradle" { + t.Fatalf("expected fallback to gradle binary when no wrapper found") + } +} + +func TestResolveGradleTreeFailureIsFatal(t *testing.T) { + dir := t.TempDir() + runner := &fakeRunner{err: fmt.Errorf("exit status 1"), result: CommandResult{Stderr: "Configuration 'runtimeClasspath' not found"}} + + _, _, err := resolveGradleTree(context.Background(), Project{Manager: ManagerGradle, Dir: dir, File: filepath.Join(dir, "build.gradle")}, Options{Runner: runner, MaxDepth: 0}) + if err == nil { + t.Fatal("expected fatal tool error") + } + var te toolError + if !errors.As(err, &te) { + t.Fatalf("expected toolError, got %T: %v", err, err) + } + if !strings.Contains(err.Error(), gradleDefaultConfiguration) { + t.Fatalf("error should name the configuration, got %v", err) + } +} diff --git a/deps/helm_credentials.go b/deps/helm_credentials.go new file mode 100644 index 0000000..c8a89a6 --- /dev/null +++ b/deps/helm_credentials.go @@ -0,0 +1,112 @@ +package deps + +import ( + "crypto/tls" + "crypto/x509" + "net/http" + "os" + "strings" + + "github.com/goccy/go-yaml" + "helm.sh/helm/v3/pkg/helmpath" +) + +// helmRepoCreds mirrors one entry of Helm's repositories.yaml, carrying the +// authentication a private chart repository needs. +type helmRepoCreds struct { + Name string `yaml:"name"` + URL string `yaml:"url"` + Username string `yaml:"username"` + Password string `yaml:"password"` + CertFile string `yaml:"certFile"` + KeyFile string `yaml:"keyFile"` + CAFile string `yaml:"caFile"` + InsecureSkipTLSVerify bool `yaml:"insecure_skip_tls_verify"` +} + +type helmRepoFile struct { + Repositories []helmRepoCreds `yaml:"repositories"` +} + +// helmCredentials reuses the user's `helm repo add` logins so chart discovery +// authenticates to private repositories exactly like the helm CLI. +type helmCredentials struct { + repos []helmRepoCreds +} + +// loadHelmCredentials reads repositories.yaml from $HELM_REPOSITORY_CONFIG or +// Helm's default config path. A missing or unreadable file yields no +// credentials (public repositories keep working). +func loadHelmCredentials() *helmCredentials { + path := os.Getenv("HELM_REPOSITORY_CONFIG") + if path == "" { + path = helmpath.ConfigPath("repositories.yaml") + } + return newHelmCredentials(path) +} + +func newHelmCredentials(path string) *helmCredentials { + data, err := os.ReadFile(path) + if err != nil { + return &helmCredentials{} + } + var f helmRepoFile + if err := yaml.Unmarshal(data, &f); err != nil { + return &helmCredentials{} + } + return &helmCredentials{repos: f.Repositories} +} + +// match returns the credentials for the repository whose URL is the longest +// prefix of url (so index.yaml/.tgz fetches under a repo inherit its login). +func (h *helmCredentials) match(url string) (helmRepoCreds, bool) { + best := -1 + var found helmRepoCreds + for _, r := range h.repos { + base := strings.TrimSuffix(r.URL, "/") + if base == "" || (url != base && !strings.HasPrefix(url, base+"/")) { + continue + } + if len(base) > best { + best, found = len(base), r + } + } + return found, best >= 0 +} + +// authorize applies the matched repository's basic-auth and TLS settings to req, +// returning the http client to use (a custom one only when TLS options apply). +func (h *helmCredentials) authorize(req *http.Request) *http.Client { + if h == nil { + return http.DefaultClient + } + cred, ok := h.match(req.URL.String()) + if !ok { + return http.DefaultClient + } + if cred.Username != "" { + req.SetBasicAuth(cred.Username, cred.Password) + } + return httpClientForCreds(cred) +} + +func httpClientForCreds(cred helmRepoCreds) *http.Client { + if cred.CAFile == "" && cred.CertFile == "" && !cred.InsecureSkipTLSVerify { + return http.DefaultClient + } + tlsCfg := &tls.Config{InsecureSkipVerify: cred.InsecureSkipTLSVerify} //nolint:gosec // honors the user's helm repo setting + if cred.CAFile != "" { + if pem, err := os.ReadFile(cred.CAFile); err == nil { + pool := x509.NewCertPool() + if pool.AppendCertsFromPEM(pem) { + tlsCfg.RootCAs = pool + } + } + } + if cred.CertFile != "" && cred.KeyFile != "" { + if cert, err := tls.LoadX509KeyPair(cred.CertFile, cred.KeyFile); err == nil { + tlsCfg.Certificates = []tls.Certificate{cert} + } + } + return &http.Client{Transport: &http.Transport{TLSClientConfig: tlsCfg}} +} diff --git a/deps/helm_credentials_test.go b/deps/helm_credentials_test.go new file mode 100644 index 0000000..af00f45 --- /dev/null +++ b/deps/helm_credentials_test.go @@ -0,0 +1,78 @@ +package deps + +import ( + "context" + "net/http" + "net/http/httptest" + "testing" + "time" +) + +func TestHelmCredentialsMatchLongestPrefix(t *testing.T) { + creds := &helmCredentials{repos: []helmRepoCreds{ + {Name: "broad", URL: "https://charts.example.com"}, + {Name: "narrow", URL: "https://charts.example.com/team/", Username: "u"}, + }} + got, ok := creds.match("https://charts.example.com/team/index.yaml") + if !ok || got.Name != "narrow" { + t.Fatalf("expected longest-prefix match 'narrow', got %q (ok=%v)", got.Name, ok) + } + if other, ok := creds.match("https://unrelated.example.com/index.yaml"); ok { + t.Fatalf("unrelated URL should not match, got %q", other.Name) + } +} + +func TestCacheFetchSendsHelmBasicAuth(t *testing.T) { + var gotUser, gotPass string + var hadAuth bool + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotUser, gotPass, hadAuth = r.BasicAuth() + if !hadAuth { + w.WriteHeader(http.StatusUnauthorized) + return + } + _, _ = w.Write([]byte("index-body")) + })) + defer srv.Close() + + now, _ := testClock(time.Unix(1000, 0)) + c := newTestCache(t, now) + c.get = c.httpGet // exercise the real HTTP path + c.helmAuth = &helmCredentials{repos: []helmRepoCreds{ + {Name: "private", URL: srv.URL, Username: "alice", Password: "s3cret"}, + }} + + data, err := c.Fetch(context.Background(), srv.URL+"/index.yaml", ttlIndex) + if err != nil { + t.Fatalf("fetch with credentials failed: %v", err) + } + if string(data) != "index-body" { + t.Fatalf("body = %q, want index-body", data) + } + if !hadAuth || gotUser != "alice" || gotPass != "s3cret" { + t.Fatalf("server did not receive expected basic auth (user=%q pass set=%v hadAuth=%v)", gotUser, gotPass != "", hadAuth) + } +} + +func TestCacheFetchNoAuthForUnmatchedRepo(t *testing.T) { + var hadAuth bool + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _, hadAuth = r.BasicAuth() + _, _ = w.Write([]byte("ok")) + })) + defer srv.Close() + + now, _ := testClock(time.Unix(1000, 0)) + c := newTestCache(t, now) + c.get = c.httpGet + c.helmAuth = &helmCredentials{repos: []helmRepoCreds{ + {Name: "other", URL: "https://charts.other.com", Username: "x", Password: "y"}, + }} + + if _, err := c.Fetch(context.Background(), srv.URL+"/index.yaml", ttlIndex); err != nil { + t.Fatal(err) + } + if hadAuth { + t.Fatalf("credentials must not be sent to a non-matching repository") + } +} diff --git a/deps/image_base.go b/deps/image_base.go new file mode 100644 index 0000000..fba8d0e --- /dev/null +++ b/deps/image_base.go @@ -0,0 +1,167 @@ +package deps + +import ( + "context" + "fmt" + "io/fs" + "os" + "path/filepath" + "strings" +) + +const ( + labelBaseName = "org.opencontainers.image.base.name" + labelBaseDigest = "org.opencontainers.image.base.digest" + labelSource = "org.opencontainers.image.source" +) + +// imageResolver resolves the base image(s) of a container image by combining the +// OCI base-image label with the Dockerfile FROM directives in the image's source +// repository. +type imageResolver struct { + cache RemoteCache + parser func(content string) ([]imageRef, []string) +} + +func newImageResolver(cache RemoteCache) *imageResolver { + return &imageResolver{cache: cache, parser: parseDockerfileFrom} +} + +// baseImages returns the distinct base images of ref. It never returns a hard +// error: remote failures are reported as warnings so recursion degrades per +// branch. +func (r *imageResolver) baseImages(ctx context.Context, ref string) (bases []imageRef, warnings []string) { + cfg, err := r.cache.ImageConfig(ctx, ref) + if err != nil { + return nil, []string{fmt.Sprintf("image %s: %s", ref, err)} + } + + seen := map[string]bool{} + add := func(b imageRef) { + key := b.Name + ":" + b.Version + "@" + b.Digest + if b.Name == "" || seen[key] { + return + } + seen[key] = true + bases = append(bases, b) + } + + // Signal A: explicit OCI base-image label. + if name := cfg.Labels[labelBaseName]; name != "" { + b := parseImageRef(name) + if b.Digest == "" { + b.Digest = cfg.Labels[labelBaseDigest] + } + add(b) + } + + // Signal B: source repo Dockerfile FROM. + source := cfg.Labels[labelSource] + if source == "" { + source = sourceRepoHeuristic(ref) + } + if source != "" { + dbases, dwarn := r.dockerfileBases(ctx, ref, source) + warnings = append(warnings, dwarn...) + for _, b := range dbases { + add(b) + } + } + + if len(bases) == 0 && len(warnings) == 0 { + warnings = append(warnings, fmt.Sprintf("image %s: no base image resolvable (no base/source label, no known registry heuristic)", ref)) + } + return bases, warnings +} + +func (r *imageResolver) dockerfileBases(ctx context.Context, ref, source string) ([]imageRef, []string) { + url := normalizeGitURL(source) + if url == "" { + return nil, []string{fmt.Sprintf("image %s: unrecognized source %q", ref, source)} + } + dir, err := r.cache.GitRepo(ctx, url, parseImageRef(ref).Version) + if err != nil { + return nil, []string{fmt.Sprintf("image %s: clone %s: %s", ref, url, err)} + } + path, warnings := findDockerfile(dir) + if path == "" { + return nil, []string{fmt.Sprintf("image %s: no Dockerfile in %s", ref, source)} + } + data, err := os.ReadFile(path) + if err != nil { + return nil, append(warnings, fmt.Sprintf("image %s: reading Dockerfile: %s", ref, err)) + } + bases, pwarn := r.parser(string(data)) + return bases, append(warnings, pwarn...) +} + +// sourceRepoHeuristic maps a registry path to a probable git source URL for +// registries that mirror their org/repo layout from GitHub. +func sourceRepoHeuristic(ref string) string { + name := stripImageVersion(ref) + for _, host := range []string{"ghcr.io/", "quay.io/"} { + if strings.HasPrefix(name, host) { + parts := strings.Split(strings.TrimPrefix(name, host), "/") + if len(parts) >= 2 { + return "https://github.com/" + parts[0] + "/" + parts[1] + } + } + } + return "" +} + +// normalizeGitURL turns an OCI source label into a cloneable URL. +func normalizeGitURL(source string) string { + source = strings.TrimSpace(source) + source = strings.TrimPrefix(source, "git+") + switch { + case source == "": + return "" + case strings.HasPrefix(source, "http://"), strings.HasPrefix(source, "https://"), strings.HasPrefix(source, "git@"): + return strings.TrimSuffix(source, ".git") + case strings.HasPrefix(source, "github.com/"), strings.HasPrefix(source, "gitlab.com/"): + return "https://" + strings.TrimSuffix(source, ".git") + default: + return "" + } +} + +// findDockerfile returns the repo-root Dockerfile if present, else the first +// Dockerfile found in a shallow walk, warning when several exist. +func findDockerfile(root string) (string, []string) { + if p := filepath.Join(root, "Dockerfile"); fileExists(p) { + return p, nil + } + var found []string + _ = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return nil + } + if d.IsDir() { + if d.Name() == ".git" || ignoredDirs[d.Name()] { + return filepath.SkipDir + } + return nil + } + if d.Name() == "Dockerfile" || strings.HasPrefix(d.Name(), "Dockerfile.") { + found = append(found, path) + } + return nil + }) + if len(found) == 0 { + return "", nil + } + if len(found) > 1 { + rels := make([]string, len(found)) + for i, f := range found { + rels[i], _ = filepath.Rel(root, f) + } + return found[0], []string{fmt.Sprintf("multiple Dockerfiles found, using %s (others: %s)", rels[0], strings.Join(rels[1:], ", "))} + } + return found[0], nil +} + +func fileExists(path string) bool { + info, err := os.Stat(path) + return err == nil && !info.IsDir() +} diff --git a/deps/manager/gomod/warm.go b/deps/manager/gomod/warm.go new file mode 100644 index 0000000..11ad1e3 --- /dev/null +++ b/deps/manager/gomod/warm.go @@ -0,0 +1,95 @@ +// Package gomod warms the Go module and build caches for a single module. +// +// It is named gomod rather than go because a directory named "go" reads badly at +// import sites. +package gomod + +import ( + "fmt" + + "github.com/flanksource/repomap/deps/manifest" +) + +// ModulePath is the synthetic module the scratch project declares. It is never +// published, but it must be a valid module path for `go mod init` to accept it. +const ModulePath = "repomap.local/cachewarm" + +// Warmer drives the go toolchain against a throwaway module that requires the +// target, populating GOMODCACHE and — with Build — GOCACHE. +type Warmer struct{} + +func (Warmer) Manager() manifest.Manager { return manifest.ManagerGo } + +func (Warmer) Binary() string { return "go" } + +// Probe returns nil: the go commands used here have been stable for many +// releases, so no runtime version check is needed. +func (Warmer) Probe() *manifest.Command { return nil } + +func (Warmer) Steps(req manifest.WarmRequest, _ string) ([]manifest.Step, error) { + switch { + case req.Dir == "": + return nil, fmt.Errorf("go warming needs a scratch directory") + case req.Name == "": + return nil, fmt.Errorf("go warming needs a module path") + case req.Version == "": + return nil, fmt.Errorf("go warming needs a version for %s", req.Name) + } + + // The /... package pattern, rather than the bare module, is what records the + // go.sum entries needed to *build* every package in the module. Resolving the + // module alone records only enough to reference it, which leaves a later + // offline build short of its dependencies. + packages := req.Name + "/..." + + steps := []manifest.Step{ + goStep("init", req, false, "mod", "init", ModulePath), + goStep("resolve", req, false, "get", packages+"@"+req.Version), + // The synthetic module imports nothing, so a bare `go mod download` would + // have no packages to work from. The `all` pattern materialises zips for + // the whole resolved graph, which is what makes a later build offline-able. + goStep("download", req, false, "mod", "download", "all"), + } + if req.Build { + steps = append(steps, goStep("build", req, false, "build", packages)) + } + if req.Verify { + steps = append(steps, verifyStep(req, packages)) + } + return steps, nil +} + +// verifyStep replays the most demanding work already done, with the proxy +// disabled so any cache miss is a hard failure rather than a silent refetch. +// With Build there are compiled packages to reproduce; without it, the strongest +// available claim is that every module zip is already resident. +func verifyStep(req manifest.WarmRequest, packages string) manifest.Step { + if req.Build { + return goStep("verify", req, true, "build", packages) + } + return goStep("verify", req, true, "mod", "download", "all") +} + +func goStep(name string, req manifest.WarmRequest, offline bool, args ...string) manifest.Step { + return manifest.Step{ + Kind: manifest.StepExec, + Name: name, + Command: manifest.Command{ + Dir: req.Dir, + Name: "go", + Args: args, + Env: goEnv(offline), + }, + } +} + +// goEnv pins the module mode explicitly. GOWORK=off matters most: without it a +// scratch dir that happens to sit inside a go.work tree silently joins that +// workspace and warms the wrong module graph. +func goEnv(offline bool) []string { + env := []string{"GOWORK=off", "GOFLAGS=-mod=mod"} + if offline { + env = append(env, "GOPROXY=off") + } + return env +} diff --git a/deps/manager/gomod/warm_test.go b/deps/manager/gomod/warm_test.go new file mode 100644 index 0000000..e0c350a --- /dev/null +++ b/deps/manager/gomod/warm_test.go @@ -0,0 +1,159 @@ +package gomod + +import ( + "strings" + "testing" + + "github.com/flanksource/repomap/deps/manifest" +) + +// formatSteps renders each step as "name: argv [env]" so a single assertion pins +// the command sequence, the exact arguments, and the environment together. +func formatSteps(t *testing.T, steps []manifest.Step) []string { + t.Helper() + out := make([]string, 0, len(steps)) + for _, step := range steps { + if step.Kind != manifest.StepExec { + t.Fatalf("go warming should only produce exec steps, got %q for %q", step.Kind, step.Name) + } + out = append(out, step.Name+": "+step.Command.String()+" ["+strings.Join(step.Command.Env, " ")+"]") + } + return out +} + +func TestStepsPerFlagCombination(t *testing.T) { + const ( + online = "[GOWORK=off GOFLAGS=-mod=mod]" + offline = "[GOWORK=off GOFLAGS=-mod=mod GOPROXY=off]" + module = "github.com/acme/lib" + version = "v1.2.3" + ) + cases := []struct { + name string + build bool + verif bool + want []string + }{ + { + name: "download only", + want: []string{ + "init: go mod init " + ModulePath + " " + online, + "resolve: go get github.com/acme/lib/...@v1.2.3 " + online, + "download: go mod download all " + online, + }, + }, + { + name: "build compiles every package", + build: true, + want: []string{ + "init: go mod init " + ModulePath + " " + online, + "resolve: go get github.com/acme/lib/...@v1.2.3 " + online, + "download: go mod download all " + online, + "build: go build github.com/acme/lib/... " + online, + }, + }, + { + // Without --build there is nothing compiled to replay, so the offline + // proof is that every module zip is already resident. + name: "verify without build replays the download", + verif: true, + want: []string{ + "init: go mod init " + ModulePath + " " + online, + "resolve: go get github.com/acme/lib/...@v1.2.3 " + online, + "download: go mod download all " + online, + "verify: go mod download all " + offline, + }, + }, + { + name: "verify with build replays the build", + build: true, + verif: true, + want: []string{ + "init: go mod init " + ModulePath + " " + online, + "resolve: go get github.com/acme/lib/...@v1.2.3 " + online, + "download: go mod download all " + online, + "build: go build github.com/acme/lib/... " + online, + "verify: go build github.com/acme/lib/... " + offline, + }, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + steps, err := (Warmer{}).Steps(manifest.WarmRequest{ + Dir: "/scratch", + Name: module, + Version: version, + Build: tc.build, + Verify: tc.verif, + }, "") + if err != nil { + t.Fatal(err) + } + got := formatSteps(t, steps) + if strings.Join(got, "\n") != strings.Join(tc.want, "\n") { + t.Fatalf("steps mismatch\n got: %s\nwant: %s", strings.Join(got, "\n "), strings.Join(tc.want, "\n ")) + } + }) + } +} + +// GOPROXY=off must never leak onto a warming step, or the warm would fail on a +// cold cache instead of populating it. +func TestOnlyVerifyStepDisablesTheProxy(t *testing.T) { + steps, err := (Warmer{}).Steps(manifest.WarmRequest{ + Dir: "/scratch", Name: "github.com/acme/lib", Version: "v1.0.0", Build: true, Verify: true, + }, "") + if err != nil { + t.Fatal(err) + } + for _, step := range steps { + offline := strings.Contains(strings.Join(step.Command.Env, " "), "GOPROXY=off") + if offline != (step.Name == "verify") { + t.Errorf("step %q: GOPROXY=off present = %v, want %v", step.Name, offline, step.Name == "verify") + } + } +} + +// Every step must run inside the scratch project, otherwise go would resolve +// against whatever module happens to contain the process working directory. +func TestStepsRunInTheScratchDir(t *testing.T) { + const dir = "/tmp/repomap-cache-warm-xyz" + steps, err := (Warmer{}).Steps(manifest.WarmRequest{ + Dir: dir, Name: "github.com/acme/lib", Version: "v1.0.0", Build: true, Verify: true, + }, "") + if err != nil { + t.Fatal(err) + } + for _, step := range steps { + if step.Command.Dir != dir { + t.Errorf("step %q dir = %q, want %q", step.Name, step.Command.Dir, dir) + } + } +} + +func TestStepsRejectsIncompleteRequest(t *testing.T) { + cases := []struct { + name string + request manifest.WarmRequest + }{ + {name: "no module path", request: manifest.WarmRequest{Dir: "/scratch", Version: "v1.0.0"}}, + {name: "no version", request: manifest.WarmRequest{Dir: "/scratch", Name: "github.com/acme/lib"}}, + {name: "no dir", request: manifest.WarmRequest{Name: "github.com/acme/lib", Version: "v1.0.0"}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if _, err := (Warmer{}).Steps(tc.request, ""); err == nil { + t.Fatal("expected an error for an incomplete request") + } + }) + } +} + +func TestWarmerIdentity(t *testing.T) { + if got := (Warmer{}).Manager(); got != manifest.ManagerGo { + t.Errorf("Manager() = %q, want %q", got, manifest.ManagerGo) + } + if probe := (Warmer{}).Probe(); probe != nil { + t.Errorf("Probe() = %v, want nil: go needs no runtime version check", probe) + } +} diff --git a/deps/manager/node/warm.go b/deps/manager/node/warm.go new file mode 100644 index 0000000..05b56e2 --- /dev/null +++ b/deps/manager/node/warm.go @@ -0,0 +1,83 @@ +// Package node holds the warming logic npm and pnpm share: generating the +// single-dependency package.json and assembling the install/verify step +// sequence. The npm and pnpm packages supply only the argv that differs. +package node + +import ( + "encoding/json" + "fmt" + + "github.com/flanksource/repomap/deps/manifest" +) + +// ProjectName is the synthetic package the scratch project declares. It is +// marked private so the registry client never treats it as publishable. +const ProjectName = "repomap-cache-warm" + +// Commands is how one node package manager spells the operations warming needs. +type Commands struct { + Binary string + // Install is the base argv that downloads and links the dependency. + Install []string + // IgnoreScripts suppresses dependency lifecycle scripts. It is applied unless + // the caller asked to build, since compiling native addons is the point of + // building. + IgnoreScripts string + // BuildArgs are appended when building. pnpm uses this for its version-gated + // lifecycle-script allowlist; npm needs nothing. + BuildArgs []string + // Offline is the full argv for the replay that proves the cache is complete. + Offline []string +} + +// Manifest renders the scratch package.json. This is the one place repomap +// generates a package manifest rather than editing an existing one. +func Manifest(name, version string) ([]byte, error) { + return json.MarshalIndent(struct { + Name string `json:"name"` + Version string `json:"version"` + Private bool `json:"private"` + Dependencies map[string]string `json:"dependencies"` + }{ + Name: ProjectName, + Version: "0.0.0", + Private: true, + Dependencies: map[string]string{name: version}, + }, "", " ") +} + +func Steps(req manifest.WarmRequest, cmds Commands) ([]manifest.Step, error) { + switch { + case req.Dir == "": + return nil, fmt.Errorf("%s warming needs a scratch directory", cmds.Binary) + case req.Name == "": + return nil, fmt.Errorf("%s warming needs a package name", cmds.Binary) + case req.Version == "": + return nil, fmt.Errorf("%s warming needs a version for %s", cmds.Binary, req.Name) + } + content, err := Manifest(req.Name, req.Version) + if err != nil { + return nil, err + } + + install := append([]string{}, cmds.Install...) + if req.Build { + install = append(install, cmds.BuildArgs...) + } else if cmds.IgnoreScripts != "" { + install = append(install, cmds.IgnoreScripts) + } + + steps := []manifest.Step{ + {Kind: manifest.StepWrite, Name: "manifest", Path: "package.json", Content: content}, + {Kind: manifest.StepExec, Name: "download", Command: manifest.Command{Dir: req.Dir, Name: cmds.Binary, Args: install}}, + } + if req.Verify { + // Installing over a populated node_modules is a no-op, so the tree has to + // go before the offline replay can prove the cache holds the packages. + steps = append(steps, + manifest.Step{Kind: manifest.StepRemove, Name: "clean", Path: "node_modules"}, + manifest.Step{Kind: manifest.StepExec, Name: "verify", Command: manifest.Command{Dir: req.Dir, Name: cmds.Binary, Args: cmds.Offline}}, + ) + } + return steps, nil +} diff --git a/deps/manager/node/warm_test.go b/deps/manager/node/warm_test.go new file mode 100644 index 0000000..542026a --- /dev/null +++ b/deps/manager/node/warm_test.go @@ -0,0 +1,127 @@ +package node + +import ( + "strings" + "testing" + + "github.com/flanksource/repomap/deps/manifest" +) + +func TestManifestDeclaresExactlyOneDependency(t *testing.T) { + cases := []struct { + name string + pkg string + version string + want string + }{ + { + name: "plain package", pkg: "left-pad", version: "1.3.0", + want: `{ + "name": "repomap-cache-warm", + "version": "0.0.0", + "private": true, + "dependencies": { + "left-pad": "1.3.0" + } +}`, + }, + { + // A scoped name must survive verbatim as the dependency key. + name: "scoped package", pkg: "@flanksource/clicky-ui", version: "^2.1.0", + want: `{ + "name": "repomap-cache-warm", + "version": "0.0.0", + "private": true, + "dependencies": { + "@flanksource/clicky-ui": "^2.1.0" + } +}`, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, err := Manifest(tc.pkg, tc.version) + if err != nil { + t.Fatal(err) + } + if string(got) != tc.want { + t.Fatalf("manifest mismatch\n got: %s\nwant: %s", got, tc.want) + } + }) + } +} + +func testCommands() Commands { + return Commands{ + Binary: "fakepm", + Install: []string{"install"}, + IgnoreScripts: "--ignore-scripts", + Offline: []string{"install", "--offline"}, + } +} + +func TestStepsWriteManifestBeforeInstalling(t *testing.T) { + steps, err := Steps(manifest.WarmRequest{Dir: "/scratch", Name: "left-pad", Version: "1.3.0"}, testCommands()) + if err != nil { + t.Fatal(err) + } + want := []string{ + "manifest: write package.json", + "download: exec fakepm install --ignore-scripts", + } + if got := manifest.FormatSteps(steps); strings.Join(got, "\n") != strings.Join(want, "\n") { + t.Fatalf("steps mismatch\n got: %v\nwant: %v", got, want) + } +} + +func TestBuildDropsIgnoreScriptsAndAppendsBuildArgs(t *testing.T) { + cmds := testCommands() + cmds.BuildArgs = []string{"--allow-builds"} + steps, err := Steps(manifest.WarmRequest{Dir: "/scratch", Name: "left-pad", Version: "1.3.0", Build: true}, cmds) + if err != nil { + t.Fatal(err) + } + want := []string{ + "manifest: write package.json", + "download: exec fakepm install --allow-builds", + } + if got := manifest.FormatSteps(steps); strings.Join(got, "\n") != strings.Join(want, "\n") { + t.Fatalf("steps mismatch\n got: %v\nwant: %v", got, want) + } +} + +// Reinstalling over a populated node_modules is a no-op, so without the removal +// the offline replay would pass without proving the cache holds anything. +func TestVerifyRemovesNodeModulesBeforeReplaying(t *testing.T) { + steps, err := Steps(manifest.WarmRequest{Dir: "/scratch", Name: "left-pad", Version: "1.3.0", Verify: true}, testCommands()) + if err != nil { + t.Fatal(err) + } + want := []string{ + "manifest: write package.json", + "download: exec fakepm install --ignore-scripts", + "clean: remove node_modules", + "verify: exec fakepm install --offline", + } + if got := manifest.FormatSteps(steps); strings.Join(got, "\n") != strings.Join(want, "\n") { + t.Fatalf("steps mismatch\n got: %v\nwant: %v", got, want) + } +} + +func TestStepsRejectsIncompleteRequest(t *testing.T) { + cases := []struct { + name string + request manifest.WarmRequest + }{ + {name: "no package name", request: manifest.WarmRequest{Dir: "/scratch", Version: "1.3.0"}}, + {name: "no version", request: manifest.WarmRequest{Dir: "/scratch", Name: "left-pad"}}, + {name: "no dir", request: manifest.WarmRequest{Name: "left-pad", Version: "1.3.0"}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if _, err := Steps(tc.request, testCommands()); err == nil { + t.Fatal("expected an error for an incomplete request") + } + }) + } +} diff --git a/deps/manager/npm/warm.go b/deps/manager/npm/warm.go new file mode 100644 index 0000000..c0b487e --- /dev/null +++ b/deps/manager/npm/warm.go @@ -0,0 +1,28 @@ +// Package npm warms the npm cache for a single package. +package npm + +import ( + "github.com/flanksource/repomap/deps/manager/node" + "github.com/flanksource/repomap/deps/manifest" +) + +type Warmer struct{} + +func (Warmer) Manager() manifest.Manager { return manifest.ManagerNPM } + +func (Warmer) Binary() string { return "npm" } + +// Probe returns nil: npm has kept its lifecycle-script default, so no runtime +// version check is needed. +func (Warmer) Probe() *manifest.Command { return nil } + +func (Warmer) Steps(req manifest.WarmRequest, _ string) ([]manifest.Step, error) { + return node.Steps(req, node.Commands{ + Binary: "npm", + Install: []string{"install"}, + IgnoreScripts: "--ignore-scripts", + // ci rather than install: the lockfile the download step wrote makes it the + // stricter replay, and it refuses to reach the network for anything missing. + Offline: []string{"ci", "--offline"}, + }) +} diff --git a/deps/manager/npm/warm_test.go b/deps/manager/npm/warm_test.go new file mode 100644 index 0000000..7fc5f5f --- /dev/null +++ b/deps/manager/npm/warm_test.go @@ -0,0 +1,69 @@ +package npm + +import ( + "strings" + "testing" + + "github.com/flanksource/repomap/deps/manifest" +) + +func TestSteps(t *testing.T) { + cases := []struct { + name string + build bool + verif bool + want []string + }{ + { + name: "download suppresses lifecycle scripts", + want: []string{ + "manifest: write package.json", + "download: exec npm install --ignore-scripts", + }, + }, + { + // Compiling native addons is the point of building, so the suppression + // has to come off. + name: "build allows lifecycle scripts", + build: true, + want: []string{ + "manifest: write package.json", + "download: exec npm install", + }, + }, + { + // npm ci over npm install: the lockfile written by the download step + // makes it the stricter replay. + name: "verify replays from the lockfile offline", + verif: true, + want: []string{ + "manifest: write package.json", + "download: exec npm install --ignore-scripts", + "clean: remove node_modules", + "verify: exec npm ci --offline", + }, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + steps, err := (Warmer{}).Steps(manifest.WarmRequest{ + Dir: "/scratch", Name: "left-pad", Version: "1.3.0", Build: tc.build, Verify: tc.verif, + }, "") + if err != nil { + t.Fatal(err) + } + if got := manifest.FormatSteps(steps); strings.Join(got, "\n") != strings.Join(tc.want, "\n") { + t.Fatalf("steps mismatch\n got: %v\nwant: %v", got, tc.want) + } + }) + } +} + +func TestWarmerIdentity(t *testing.T) { + if got := (Warmer{}).Manager(); got != manifest.ManagerNPM { + t.Errorf("Manager() = %q, want %q", got, manifest.ManagerNPM) + } + if probe := (Warmer{}).Probe(); probe != nil { + t.Errorf("Probe() = %v, want nil: npm needs no runtime version check", probe) + } +} diff --git a/deps/manager/pnpm/warm.go b/deps/manager/pnpm/warm.go new file mode 100644 index 0000000..c5ddc8f --- /dev/null +++ b/deps/manager/pnpm/warm.go @@ -0,0 +1,66 @@ +// Package pnpm warms the pnpm content-addressable store for a single package. +package pnpm + +import ( + "fmt" + + "github.com/Masterminds/semver/v3" + "github.com/flanksource/repomap/deps/manager/node" + "github.com/flanksource/repomap/deps/manifest" +) + +// allowAllBuilds opts every dependency into running its lifecycle scripts. pnpm +// 10.6 stopped honouring them by default — dropping --ignore-scripts is no longer +// enough — and older pnpm does not recognise the flag, hence the version gate. +const allowAllBuilds = "--config.dangerouslyAllowAllBuilds=true" + +const ( + allowAllBuildsMajor = 10 + allowAllBuildsMinor = 6 +) + +type Warmer struct{} + +func (Warmer) Manager() manifest.Manager { return manifest.ManagerPNPM } + +func (Warmer) Binary() string { return "pnpm" } + +// Probe reports the pnpm version, which decides how --build has to ask for +// lifecycle scripts. Dir is left for the orchestrator to fill in. +func (Warmer) Probe() *manifest.Command { + return &manifest.Command{Name: "pnpm", Args: []string{"--version"}} +} + +func (Warmer) Steps(req manifest.WarmRequest, probe string) ([]manifest.Step, error) { + cmds := node.Commands{ + Binary: "pnpm", + Install: []string{"install"}, + IgnoreScripts: "--ignore-scripts", + Offline: []string{"install", "--offline", "--frozen-lockfile"}, + } + // The probe only matters for building; a plain warm must not depend on it. + if req.Build { + buildArgs, err := buildArgsFor(probe) + if err != nil { + return nil, err + } + cmds.BuildArgs = buildArgs + } + return node.Steps(req, cmds) +} + +func buildArgsFor(probe string) ([]string, error) { + if probe == "" { + return nil, fmt.Errorf("--build needs the pnpm version to decide how to enable dependency builds, but `pnpm --version` reported nothing") + } + version, err := semver.NewVersion(probe) + if err != nil { + return nil, fmt.Errorf("--build needs the pnpm version to decide how to enable dependency builds, but `pnpm --version` reported %q: %w", probe, err) + } + if version.Major() > allowAllBuildsMajor || + (version.Major() == allowAllBuildsMajor && version.Minor() >= allowAllBuildsMinor) { + return []string{allowAllBuilds}, nil + } + // Before 10.6, dropping --ignore-scripts is enough on its own. + return nil, nil +} diff --git a/deps/manager/pnpm/warm_test.go b/deps/manager/pnpm/warm_test.go new file mode 100644 index 0000000..2f95c20 --- /dev/null +++ b/deps/manager/pnpm/warm_test.go @@ -0,0 +1,118 @@ +package pnpm + +import ( + "strings" + "testing" + + "github.com/flanksource/repomap/deps/manifest" +) + +func TestSteps(t *testing.T) { + cases := []struct { + name string + build bool + verif bool + probe string + want []string + }{ + { + name: "download suppresses lifecycle scripts", probe: "10.7.0", + want: []string{ + "manifest: write package.json", + "download: exec pnpm install --ignore-scripts", + }, + }, + { + name: "verify replays offline against the frozen lockfile", verif: true, probe: "10.7.0", + want: []string{ + "manifest: write package.json", + "download: exec pnpm install --ignore-scripts", + "clean: remove node_modules", + "verify: exec pnpm install --offline --frozen-lockfile", + }, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + steps, err := (Warmer{}).Steps(manifest.WarmRequest{ + Dir: "/scratch", Name: "left-pad", Version: "1.3.0", Build: tc.build, Verify: tc.verif, + }, tc.probe) + if err != nil { + t.Fatal(err) + } + if got := manifest.FormatSteps(steps); strings.Join(got, "\n") != strings.Join(tc.want, "\n") { + t.Fatalf("steps mismatch\n got: %v\nwant: %v", got, tc.want) + } + }) + } +} + +// pnpm 10.6 stopped running dependency lifecycle scripts even without +// --ignore-scripts, so on those versions building needs an explicit allowlist +// flag that older pnpm does not recognise. +func TestBuildFlagIsGatedOnTheProbedVersion(t *testing.T) { + cases := []struct { + version string + want string + }{ + {version: "9.12.0", want: "download: exec pnpm install"}, + {version: "10.5.9", want: "download: exec pnpm install"}, + {version: "10.6.0", want: "download: exec pnpm install --config.dangerouslyAllowAllBuilds=true"}, + {version: "10.7.1", want: "download: exec pnpm install --config.dangerouslyAllowAllBuilds=true"}, + {version: "11.0.0", want: "download: exec pnpm install --config.dangerouslyAllowAllBuilds=true"}, + } + for _, tc := range cases { + t.Run(tc.version, func(t *testing.T) { + steps, err := (Warmer{}).Steps(manifest.WarmRequest{ + Dir: "/scratch", Name: "left-pad", Version: "1.3.0", Build: true, + }, tc.version) + if err != nil { + t.Fatal(err) + } + got := manifest.FormatSteps(steps) + if len(got) != 2 || got[1] != tc.want { + t.Fatalf("pnpm %s download step = %q, want %q", tc.version, got[len(got)-1], tc.want) + } + }) + } +} + +// Guessing the flag set would either skip the builds the user asked for or pass +// an argument older pnpm rejects, so an unreadable probe is a hard failure. +func TestBuildRejectsAnUnreadableProbe(t *testing.T) { + for _, probe := range []string{"", "not-a-version"} { + if _, err := (Warmer{}).Steps(manifest.WarmRequest{ + Dir: "/scratch", Name: "left-pad", Version: "1.3.0", Build: true, + }, probe); err == nil { + t.Errorf("probe %q: expected an error when --build cannot determine the pnpm version", probe) + } + } +} + +// Without --build the version is irrelevant, so a missing probe must not block a +// plain warm. +func TestDownloadIgnoresTheProbe(t *testing.T) { + steps, err := (Warmer{}).Steps(manifest.WarmRequest{ + Dir: "/scratch", Name: "left-pad", Version: "1.3.0", + }, "") + if err != nil { + t.Fatal(err) + } + want := []string{"manifest: write package.json", "download: exec pnpm install --ignore-scripts"} + if got := manifest.FormatSteps(steps); strings.Join(got, "\n") != strings.Join(want, "\n") { + t.Fatalf("steps mismatch\n got: %v\nwant: %v", got, want) + } +} + +func TestWarmerIdentity(t *testing.T) { + if got := (Warmer{}).Manager(); got != manifest.ManagerPNPM { + t.Errorf("Manager() = %q, want %q", got, manifest.ManagerPNPM) + } + probe := (Warmer{}).Probe() + if probe == nil { + t.Fatal("Probe() = nil, want pnpm --version: the build flag depends on it") + } + if got := probe.String(); got != "pnpm --version" { + t.Errorf("Probe() = %q, want %q", got, "pnpm --version") + } +} diff --git a/deps/manifest/command.go b/deps/manifest/command.go new file mode 100644 index 0000000..19f460c --- /dev/null +++ b/deps/manifest/command.go @@ -0,0 +1,46 @@ +package manifest + +import ( + "context" + "os" + "os/exec" + "strings" +) + +type Command struct { + Dir string + Name string + Args []string + Env []string +} + +// String renders the command as it would be typed, so failures can name the +// exact invocation to rerun. +func (c Command) String() string { + return strings.Join(append([]string{c.Name}, c.Args...), " ") +} + +type CommandResult struct { + Stdout string + Stderr string +} + +type CommandRunner interface { + Run(ctx context.Context, cmd Command) (CommandResult, error) +} + +type ExecRunner struct{} + +func (ExecRunner) Run(ctx context.Context, spec Command) (CommandResult, error) { + cmd := exec.CommandContext(ctx, spec.Name, spec.Args...) + cmd.Dir = spec.Dir + if len(spec.Env) > 0 { + cmd.Env = append(os.Environ(), spec.Env...) + } + out, err := cmd.Output() + result := CommandResult{Stdout: string(out)} + if exit, ok := err.(*exec.ExitError); ok { + result.Stderr = string(exit.Stderr) + } + return result, err +} diff --git a/deps/manifest/manager.go b/deps/manifest/manager.go new file mode 100644 index 0000000..edf4716 --- /dev/null +++ b/deps/manifest/manager.go @@ -0,0 +1,17 @@ +// Package manifest holds the dependency-manager taxonomy and the process +// execution seam shared between the deps orchestrator and the per-manager +// packages under deps/manager. It is a leaf: nothing here imports deps, so the +// manager packages can depend on it without creating an import cycle. +package manifest + +type Manager string + +const ( + ManagerGo Manager = "go" + ManagerMaven Manager = "maven" + ManagerGradle Manager = "gradle" + ManagerNPM Manager = "npm" + ManagerPNPM Manager = "pnpm" + ManagerImage Manager = "image" + ManagerHelm Manager = "helm" +) diff --git a/deps/manifest/warm.go b/deps/manifest/warm.go new file mode 100644 index 0000000..1011239 --- /dev/null +++ b/deps/manifest/warm.go @@ -0,0 +1,82 @@ +package manifest + +// WarmRequest describes one dependency to warm into the machine's shared +// package cache. Dir is a scratch project the orchestrator has already created; +// a Warmer only decides what to run inside it. +type WarmRequest struct { + Dir string + Name string + Version string + Build bool + Verify bool +} + +type StepKind string + +const ( + StepExec StepKind = "exec" + StepWrite StepKind = "write" + StepRemove StepKind = "remove" +) + +// Step is one unit of warming work. Warming is not purely exec: node ecosystems +// need a package.json written before installing, and need node_modules removed +// before an offline replay can prove anything. +type Step struct { + Kind StepKind + Name string + Command Command // StepExec + Path string // StepWrite, StepRemove — relative to WarmRequest.Dir + Content []byte // StepWrite +} + +// Detail renders just the action, with no step name, for callers that already +// report the name separately. +func (s Step) Detail() string { + switch s.Kind { + case StepExec: + return s.Command.String() + case StepWrite: + return "write " + s.Path + case StepRemove: + return "remove " + s.Path + default: + return "unknown step kind " + string(s.Kind) + } +} + +// String renders the step as "name: kind detail", so a failure can name what was +// being attempted and tests can assert a whole sequence in one comparison. +func (s Step) String() string { + if s.Kind == StepExec { + return s.Name + ": exec " + s.Detail() + } + return s.Name + ": " + s.Detail() +} + +// FormatSteps renders a sequence one line per step. +func FormatSteps(steps []Step) []string { + out := make([]string, 0, len(steps)) + for _, step := range steps { + out = append(out, step.String()) + } + return out +} + +// Warmer decides which commands warm one ecosystem. Implementations do no I/O: +// Steps is a pure function of its arguments, which is what lets the per-manager +// packages be tested by comparing argv without a toolchain, a network, or a +// process. +// +// Probe is the escape hatch for a manager that genuinely needs runtime +// information (pnpm's lifecycle-script policy changed in 10.6). The orchestrator +// runs it and feeds the trimmed stdout back into Steps as an ordinary input, +// keeping Steps pure. +type Warmer interface { + Manager() Manager + // Binary is the executable that must be on PATH, so the orchestrator can + // fail before it creates a scratch directory. + Binary() string + Probe() *Command + Steps(req WarmRequest, probe string) ([]Step, error) +} diff --git a/deps/maven.go b/deps/maven.go new file mode 100644 index 0000000..9c86fb2 --- /dev/null +++ b/deps/maven.go @@ -0,0 +1,101 @@ +package deps + +import ( + "encoding/xml" + "os" + "strings" +) + +func resolveMavenManifest(project Project) (*Node, []Warning, error) { + root, err := parseMavenPOM(project.File) + if err != nil { + return nil, nil, err + } + return root, []Warning{{Manager: ManagerMaven, Project: project.Dir, Message: "offline pom.xml parsing includes direct dependencies only; resolved transitive edges are unavailable"}}, nil +} + +type pomProject struct { + XMLName xml.Name `xml:"project"` + GroupID string `xml:"groupId"` + ArtifactID string `xml:"artifactId"` + Version string `xml:"version"` + Parent pomParent `xml:"parent"` + Properties []xmlProperty `xml:"properties>*"` + Dependencies []pomDependency `xml:"dependencies>dependency"` +} + +type pomParent struct { + GroupID string `xml:"groupId"` + Version string `xml:"version"` +} + +type xmlProperty struct { + XMLName xml.Name + Value string `xml:",chardata"` +} + +type pomDependency struct { + GroupID string `xml:"groupId"` + ArtifactID string `xml:"artifactId"` + Version string `xml:"version"` + Scope string `xml:"scope"` + Optional string `xml:"optional"` +} + +func parseMavenPOM(path string) (*Node, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var pom pomProject + if err := xml.Unmarshal(data, &pom); err != nil { + return nil, err + } + group := firstNonEmpty(pom.GroupID, pom.Parent.GroupID) + version := firstNonEmpty(pom.Version, pom.Parent.Version) + props := map[string]string{ + "project.groupId": group, + "project.version": version, + "pom.groupId": group, + "pom.version": version, + "project.artifactId": pom.ArtifactID, + "pom.artifactId": pom.ArtifactID, + } + for _, prop := range pom.Properties { + props[prop.XMLName.Local] = strings.TrimSpace(prop.Value) + } + root := NewNode(ManagerMaven, group+":"+pom.ArtifactID, resolveProperty(version, props)) + root.Path = path + root.Source = "pom.xml" + for _, dep := range pom.Dependencies { + name := resolveProperty(dep.GroupID, props) + ":" + resolveProperty(dep.ArtifactID, props) + child := NewNode(ManagerMaven, name, resolveProperty(dep.Version, props)) + child.Depth = 1 + child.Direct = true + child.Scope = firstNonEmpty(dep.Scope, "compile") + child.Optional = strings.EqualFold(strings.TrimSpace(dep.Optional), "true") + root.Children = append(root.Children, child) + } + sortChildren(root) + return root, nil +} + +func resolveProperty(value string, props map[string]string) string { + value = strings.TrimSpace(value) + if strings.HasPrefix(value, "${") && strings.HasSuffix(value, "}") { + key := strings.TrimSuffix(strings.TrimPrefix(value, "${"), "}") + if props[key] != "" { + return props[key] + } + } + return value +} + +func firstNonEmpty(values ...string) string { + for _, value := range values { + if strings.TrimSpace(value) != "" { + return strings.TrimSpace(value) + } + } + return "" +} diff --git a/deps/maven_tree.go b/deps/maven_tree.go new file mode 100644 index 0000000..2423a5e --- /dev/null +++ b/deps/maven_tree.go @@ -0,0 +1,123 @@ +package deps + +import ( + "context" + "fmt" + "os" + "strings" +) + +// resolveMavenTree resolves the transitive Maven dependency graph by running +// `mvn dependency:tree` with TGF output. It fails fast with a toolError when mvn +// is missing or the command fails, suggesting --depth 1 for offline output. +func resolveMavenTree(ctx context.Context, project Project, opts Options) (*Node, []Warning, error) { + tmp, err := os.CreateTemp("", "repomap-mvn-tree-*.tgf") + if err != nil { + return nil, nil, err + } + outFile := tmp.Name() + _ = tmp.Close() + defer func() { _ = os.Remove(outFile) }() + + result, err := opts.Runner.Run(ctx, Command{ + Dir: project.Dir, + Name: "mvn", + Args: []string{"-B", "-N", "dependency:tree", "-DoutputType=tgf", "-DoutputFile=" + outFile}, + }) + if err != nil { + detail := strings.TrimSpace(result.Stderr) + if detail == "" { + detail = err.Error() + } + return nil, nil, toolError{fmt.Errorf("mvn dependency:tree failed in %s (rerun with --depth 1 for offline direct-only output): %s", project.Dir, detail)} + } + + data, err := os.ReadFile(outFile) + if err != nil { + return nil, nil, toolError{fmt.Errorf("mvn dependency:tree produced no output file for %s: %w", project.Dir, err)} + } + graph, err := parseMavenTGF(string(data)) + if err != nil { + return nil, nil, toolError{fmt.Errorf("mvn dependency:tree parse failed in %s: %w", project.Dir, err)} + } + + root := buildTreeFromEdgeGraph(edgeTreeOptions{Graph: graph, MaxDepth: opts.MaxDepth}) + if root == nil { + return nil, nil, toolError{fmt.Errorf("mvn dependency:tree produced no nodes for %s", project.Dir)} + } + root.Path = project.File + root.Source = "pom.xml" + markDirectByDepth(root) + return root, nil, nil +} + +// parseMavenTGF parses Trivial Graph Format produced by dependency:tree. The +// node section lists "