diff --git a/.spec/ARCHITECTURE.md b/.spec/ARCHITECTURE.md index 97317cec..4af37556 100644 --- a/.spec/ARCHITECTURE.md +++ b/.spec/ARCHITECTURE.md @@ -9,7 +9,7 @@ EasyP v1 separates CLI composition, module operations, generation preparation, a |-----------|--------------------------------|--------------| | internal/api | Flags, process paths/environment, adapter construction, policy command orchestration, output and exit status | Module/generation operations, configuration, rules, core, concrete adapters | | internal/modules | Dependency selection, lock validation, source roots and ownership, manifest edits, coordinated project-file updates | V1 models, Source/Cache contracts, metadata reader, filesystem | -| internal/adapters/gitmodules | Git candidates/revisions, checkout lifetime, persistent object cache and locking, tracked-file hashes and installation | System Git, module_config, module contracts, filesystem | +| internal/adapters/gitmodules | Git candidates/revisions, checkout lifetime, persistent object cache and locking, immutable materialized snapshot hashes and installation | System Git, module_config, module contracts, filesystem | | internal/adapters/module_config | Adapt repository metadata into a named module and import roots | Native protobuf.mod parser, legacy EasyP and Buf readers | | internal/migration | Preview plans, legacy conversion, integrity verification gates, backups and rollback | V1 models, module resolution, explicit migration repository, filesystem | | internal/workspace | Repository boundary and ancestor/module/config discovery | Filesystem | diff --git a/.spec/CLI.md b/.spec/CLI.md index ea77e63f..cfc6fef0 100644 --- a/.spec/CLI.md +++ b/.spec/CLI.md @@ -214,6 +214,36 @@ easyp migrate --dir . --module github.com/acme/contracts --resolve-lock --write Flag-only invocation previews without file writes unless --write is supplied. If dependency integrity checks are required, application remains blocked until --resolve-lock authorizes them. Existing native outputs are not overwritten with conflicting candidates; legacy replaced files receive .v0.bak backups and easyp.lock is retained unchanged. Legacy version metadata such as v1alpha is accepted and omitted with a warning; deps: null means an empty dependency list. Keys that the v0 parser ignored are also omitted with source-path warnings instead of being assigned invented v1 semantics, while the byte-identical v0 backup preserves them. Known fields with invalid or ambiguous values still block migration. The plan rechecks observed files before applying and rolls back ordinary write failures, but multiple replacements are not process-crash atomic. See [internal/migration/migration.go](../internal/migration/migration.go), [apply.go](../internal/migration/apply.go), and [migration review context](config/review-migration-and-polish.md). +Local directory selection may become literal generate.paths selectors +without moving sources. The plan compares whole roots, then the original +module-directory-relative paths, then complete protobuf packages for cases +that cannot be represented by literal paths. +Each candidate must preserve the exact import-name-to-physical-source map. +Omitted or empty legacy roots keep .; native manifests with no +roots use that same default. Same-package files outside selected +paths do not become generation targets, including ignored Gradle build copies. +Changed import names, hidden/vendor/nested-module boundary changes and inferred +local filters mixed with whole-module Git inputs stay blocked. The plan rechecks +sources, paths and packages before applying. See [source selection](config/package-selection.md). + +Historical easyp.lock entries may use a full SemVer tag followed by +Git's peeled-ref suffix ^{}. Migration verifies the corresponding +tag and legacy content hash before writing the native lock; it never rewrites +the historical lock or switches to HEAD. Peeled branches, abbreviated refs, +repeated suffixes and pseudo-version-shaped peeled tags are rejected. +Released v0 lock hashes cover the installed git archive '*.proto' +contents after legacy root rewrites, while the new lock covers the materialized v1 +snapshot. Migration verifies either the historical archive hash or the existing +whole-tree hash at the pinned revision. It rejects archive attributes that omit +or alter proto sources rather than silently changing their contracts. +Internal file, directory, import-root and metadata symlinks are supported. +Logical paths keep their protobuf import names. Git targets resolve only from +the pinned tree; installed snapshots contain regular resolved bytes and work +with core.symlinks=false. Selected inputs cannot escape their owner, +cross undeclared nested repositories/submodules or form cycles. Invalid unused +auxiliary links are omitted. Migration verifies historical archive contents +separately; it never substitutes the new snapshot hash for a v0 input hash. + ### easyp ls-files [flags] Lists sources from the working directory's native manifest, or a default local . root if that directory has no manifest. Unlike mod/get, this handler does not search upward for a module. The global config flag does not choose its root. diff --git a/.spec/PACKAGES.md b/.spec/PACKAGES.md index 6a1b45b0..07cc2c9e 100644 --- a/.spec/PACKAGES.md +++ b/.spec/PACKAGES.md @@ -48,7 +48,9 @@ Section-scoped extends is implemented in internal/policyinternal/adapters/gitmodules | cache.go, git.go: cache layout and Git execution; object_cache.go, object_lock_unix.go, object_lock_windows.go: reusable Git object repositories and OS locks; checkout.go, git_source.go: revision/candidate selection; download.go, files.go: installation and tracked-file hashing; identity.go: optional Git origin identity; migration.go, migration_config.go, migration_selection.go: historical revision/hash verification | +| internal/sourceview | Bounded logical resolve/open/walk over standard io/fs; local os.Root reads and alias topology checks | +| internal/adapters/gitsnapshot | Immutable Git tree/blob filesystem, SHA-1/SHA-256 repository support and host path collision checks | +| internal/adapters/gitmodules | cache.go, git.go: cache layout and Git execution; object_cache.go, object_lock_unix.go, object_lock_windows.go: reusable Git object repositories and OS locks; checkout.go, git_source.go: revision/candidate selection; download.go, files.go: installation and materialized snapshot hashing; identity.go: optional Git origin identity; migration.go, migration_config.go, migration_selection.go: historical revision/hash verification | | internal/adapters/plugin | Local, remote, built-in WASM and command executors; Info carries the explicit local execution directory | | internal/adapters/go_git | Historical project-tree walkers for breaking checks | | internal/adapters/console | Platform command execution | diff --git a/.spec/config/dependency.md b/.spec/config/dependency.md index c694b269..e69f337d 100644 --- a/.spec/config/dependency.md +++ b/.spec/config/dependency.md @@ -56,9 +56,9 @@ See [Go major version suffixes](https://go.dev/ref/mod#major-version-suffixes) a ## Lock and cache -protobuf.lock is YAML with integer version: 1 and a modules sequence. Each entry has source, version, commit and hash; the resolver sorts by source. Commits must be full 40- or 64-character hexadecimal hashes; hash is h1: plus a base64 SHA-256 digest. A commit-valued version must equal the commit. Duplicate sources, unknown fields and multiple YAML documents are rejected. The hash covers the installed regular-file snapshot using Go's directory-hash algorithm. Git symlinks and submodules are omitted, even when a symlink is materialized as a regular file by core.symlinks=false. Dependency config files must be regular Git files. Buf file filters are applied before hashing and installation; non-proto regular files remain included. Legacy lock migration retains its stricter non-regular-file rejection because it must reproduce the old hash. +protobuf.lock is YAML with integer version: 1 and a modules sequence. Each entry has source, version, commit and hash; the resolver sorts by source. Commits must be full 40- or 64-character hexadecimal hashes; hash is h1: plus a base64 SHA-256 digest. A commit-valued version must equal the commit. Duplicate sources, unknown fields and multiple YAML documents are rejected. The hash covers the installed regular-file snapshot using Go's directory-hash algorithm. Internal file, directory/root and metadata aliases resolve from the pinned Git tree and are materialized under logical names; Git pointer targets must be relative and stay within that tree. Selected unsafe, dangling, cyclic or submodule-crossing inputs fail. Raw descendant submodules remain opaque skipped boundaries. Invalid unused auxiliary links are omitted. Buf filters select logical proto paths before hashing and installation; regular non-proto files and safe non-proto aliases remain included. The sole v1 hash policy is h1/Hash1 over this snapshot. Cache identity includes source, commit and hash. Historical v0 archive reconstruction is used only by explicit migrate input verification. -The CLI resolves EASYPPATH once for a command that needs the cache (default $HOME/.easyp). gitmodules.Cache owns <EASYPPATH>/v1/git, source keys, temporary checkout names and installation paths. Installed snapshots live under its modules/<source-key>/<commit> layout; reusable bare object stores live under objects/<remote-key>, with OS locks for concurrent fetches. Application callers request cached module metadata and its physical directory; they do not assemble cache paths. +The CLI resolves EASYPPATH once for a command that needs the cache (default $HOME/.easyp). gitmodules.Cache owns <EASYPPATH>/v1/git, source keys, temporary checkout names and installation paths. Installed snapshots live under its modules/<source-key>/<snapshot-key> layout, where the snapshot key hashes commit plus content hash; reusable bare object stores live under objects/<remote-key>, with OS locks for concurrent fetches. Application callers request cached module metadata and its physical directory; they do not assemble cache paths. Cache.Install verifies locked contents, including cache hits. A newly installed snapshot is checked with the full h1: directory hash. On Darwin/Linux, later unchanged hits use a sidecar verification stamp keyed by the expected lock hash plus an inode/ctime metadata fingerprint; any metadata change invalidates the stamp and forces the full directory hash again. Content changes therefore remain detectable even when size and mtime are restored. Platforms without a strong change token keep the full-hash path. The stamp is an optimization of the per-user local cache, not an additional source of dependency identity. Cached objects support shallow pinned-commit fetches with an advertised-history fallback; neither path substitutes HEAD for an unavailable pin. Cache.Cached reads installed metadata without downloading or rewriting module contents and requires prior verification. Git credentials and transport configuration remain the responsibility of system Git. diff --git a/.spec/config/package-selection.md b/.spec/config/package-selection.md index 47895d84..af46911b 100644 --- a/.spec/config/package-selection.md +++ b/.spec/config/package-selection.md @@ -1,8 +1,8 @@ -# Exact protobuf package selection +# Protobuf package and path selection A nonempty generate.packages selects exact protobuf package names within the -modules selected by a generation project. Empty or omitted selects all source -files, preserving the existing behavior. Names follow protobuf identifier +modules selected by a generation project. Empty or omitted means all packages +within generate.paths; without either filter, all module sources participate. Names follow protobuf identifier syntax; they are not prefixes, filesystem paths, globs or expressions. ~~~yaml @@ -16,8 +16,9 @@ plugins: with_imports: true ~~~ -All files declaring a selected package participate. Matching is across the -project's selected modules, not separately required in every module. Modules +All files declaring a selected package within the selected paths participate. +Matching is across the project's selected modules, not separately required in +every module. Modules with no matching package do not execute plugins or produce empty descriptor sets. Unknown names fail before any plugin, including partially matched lists. Duplicate names are idempotent. Explicit package validation applies even to an @@ -41,6 +42,55 @@ before execution remain unchanged. Selection never modifies dependency state, proto packages, import names or a published lock. Frozen checks still verify the selected module graph; local replacements remain forbidden in frozen mode. +## Literal paths + +generate.paths selects exact module-relative files or directory +subtrees across the project's selected modules. Empty means all module sources; +. explicitly selects all. Matching is component-bounded: mcp +does not select mcp-copy. Canonical relative paths are required; +absolute paths, traversal, backslashes and globs are rejected. These selectors +are separate from plugin opts.paths, which controls output layout. +The base is each selected module's directory, not its import roots or the +generator file. For roots api, api/easyp selects files +whose compiler import names start with easyp/. + +Module entries may be strings or objects with module, optional +paths and optional packages. Module filters intersect the +global filters. Global selectors must match across selected modules; scoped +selectors must match their own module's final sources. Identical repeated +selections are idempotent (selector order and repetition do not matter); +conflicting selectors for one resolved module in a project fail before plugins. +Global filters remain available without listing a module. + +~~~yaml +version: v1 +generate: + paths: [mcp] +plugins: + - name: go + out: . + opts: {paths: source_relative} +~~~ + +Paths and packages intersect. Each selector must match a resulting source in +at least one selected module. Unknown or partially unmatched lists fail before +plugins or descriptor writes, including no-plugin projects and parents selected +with --all. Source discovery respects existing module/Buf filters; +required imports outside the selected paths still compile. No automatic Git +ignore policy is introduced. Roots, source locations and import names stay fixed. + +The migration wizard prefers exact directory paths after whole-root equality. +It falls back to complete package selectors only when paths cannot preserve a +mixed-root selection. The current import-name/physical-file maps must match, +and sources plus fixed paths/packages are rechecked before apply. Empty inferred +selection, alias/boundary changes and local filters combined with whole-module +Git generation inputs remain rejected. Future same-package build copies outside +a selected directory do not widen that directory's targets. Package fallback +previews still warn that future files declaring selected packages participate. +The sole local module is inferred from the generator's location rather than +repeated in generate.modules. Migration does not add an unconfigured +options.go.package_prefix; omitted values retain normal inheritance. + Regressions verify exact/prefix distinction, multiple files, multiple modules, no-plugin failures, invalid unselected/required sources, custom options, per- plugin imports, both descriptor modes and compiled Go output with a local diff --git a/.spec/config/review-context-and-generation.md b/.spec/config/review-context-and-generation.md index 42319758..fa189c65 100644 --- a/.spec/config/review-context-and-generation.md +++ b/.spec/config/review-context-and-generation.md @@ -20,8 +20,8 @@ easyp generate --all The all and project flags are mutually exclusive. Recursive discovery skips hidden directories, easyp_vendor, node_modules and nested Git repositories. A project in a skipped directory can still be deliberately selected with -project. Automatic selection refuses a symlink configuration file; explicit -project selection is required to use it. No name-based directory denylist is +project. Internal configuration aliases are resolved within the workspace +boundary for both automatic and explicit selection. No name-based directory denylist is claimed to establish a trust boundary: choosing all explicitly authorizes recursive generation, including command plugins in the selected tree. diff --git a/.spec/config/review-migration-and-polish.md b/.spec/config/review-migration-and-polish.md index 1259be60..c2d755c4 100644 --- a/.spec/config/review-migration-and-polish.md +++ b/.spec/config/review-migration-and-polish.md @@ -23,11 +23,27 @@ physical files and protobuf import names must remain equivalent. Legacy direct/indirect directives become require. Old full-SHA/pseudo-version pins retain their commit. A tag-only pin is accepted only after verifying the -legacy installed-tree hash. The new full tracked-repository hash is calculated +legacy installed-tree hash. The new materialized snapshot hash is calculated independently. Missing historical pins, retags and hash mismatches fail. The old easyp.lock remains byte-identical. Empty projects receive a native empty lock, so the retained old lock cannot block later normal module commands. +Internal file, directory, import-root and metadata symlinks are supported. +Logical paths keep their protobuf import names. Git targets resolve only from +the pinned tree; installed snapshots contain regular resolved bytes and work +with core.symlinks=false. Selected inputs cannot escape their owner, +cross undeclared nested repositories/submodules or form cycles. Invalid unused +auxiliary links are omitted. Migration verifies historical archive contents +separately; it never substitutes the new snapshot hash for a v0 input hash. + +Git index framing, stage and local-path validation live in the pure +internal/adapters/gitindex parser. Snapshot selection and metadata preflight +apply their own mode policies. Migration reads one index, records the selected +regular files and omitted auxiliary links, then verifies historical contents +in migration_integrity.go. FetchMigration keeps source equivalence and native +hash calculation as separate steps; no-link whole-tree proof and archive-only +proof after omitted links remain distinct. + The application transaction stages all results and .v0.bak recovery copies, checks observed contents/permissions/source scope again, and rolls back ordinary failures. Conflicting existing files and unsafe symlink destinations are refused. diff --git a/README.md b/README.md index 90f3c9c6..51c42f4b 100644 --- a/README.md +++ b/README.md @@ -254,3 +254,25 @@ Run easyp migrate in a terminal for a guided migration, or add access and applying files require separate confirmations, both defaulting to no. Explicit --module without the wizard keeps the preview-first script interface; --interactive=false disables automatic prompting. + +Legacy directory inputs can migrate without moving sources through literal +generate.paths selectors. Paths select files or directory subtrees +relative to proto import roots; they do not change protobuf.mod roots +or import access. Empty or omitted roots still mean .. For example, +paths: [mcp] keeps Gradle copies of the same package outside the +selected directory out of generation. Complete package selectors remain a +fallback for representable mixed-root configurations. The wizard proves exact +file/import equality and rechecks the fixed selection before applying it. +Historical easyp.lock entries +such as v0.4.0^{} are verified as the corresponding annotated Git tag, +including the original content hash, while the old lock remains unchanged. +The migration verifies released v0's proto archive hashes before calculating +the native materialized-snapshot hash; Git archive attributes must preserve proto paths +and contents. +Internal file, directory, import-root and metadata symlinks are supported. +Logical paths keep their protobuf import names. Git targets resolve only from +the pinned tree; installed snapshots contain regular resolved bytes and work +with core.symlinks=false. Selected inputs cannot escape their owner, +cross undeclared nested repositories/submodules or form cycles. Invalid unused +auxiliary links are omitted. Migration verifies historical archive contents +separately; it never substitutes the new snapshot hash for a v0 input hash. diff --git a/V1_RELEASE_NOTES.md b/V1_RELEASE_NOTES.md index 5a48190e..31877a66 100644 --- a/V1_RELEASE_NOTES.md +++ b/V1_RELEASE_NOTES.md @@ -57,8 +57,9 @@ The native dependency format is documented in [`.spec/config/dependency.md`](.sp The current command behavior is documented in [`.spec/CLI.md`](.spec/CLI.md): - generation discovery is nearest-project by default and recursive only with `--all`; -- `generate.modules` selects module identities or workspace module paths; +- `generate.modules` selects module identities or workspace module paths; object entries add per-module paths/packages, intersected with global filters; - `generate.packages` selects exact protobuf package names; +- `generate.paths` selects literal module-directory-relative files or directory subtrees, intersecting packages; it remains available without a module list; - dependencies are available for imports but are not automatically generation targets; - `with_imports` is per-plugin and does not change descriptor-export `--include_imports` semantics. - source-relative Go output follows the effective descriptor `go_package`, including disable rules, overrides, and path markers; plugin-controlled import layouts stay intact. See [generation details](.spec/config/review-generation-and-baselines.md). @@ -67,4 +68,24 @@ The current command behavior is documented in [`.spec/CLI.md`](.spec/CLI.md): `easyp migrate` is the supported v0-to-v1 transition path. It accepts the documented legacy compatibility metadata, preserves legacy inputs until apply succeeds, and does not silently reinterpret a pilot-RFC file as a native v1 manifest. +The wizard preserves legacy directory selections through literal +generate.paths, retaining root ., source locations and +import names. Paths select files or component-bounded directory subtrees and +intersect optional package selectors. Same-package build copies outside the +selected path stay out of generation. Complete packages remain a mixed-root +fallback. Unknown selectors fail before plugins, including no-plugin parents +selected with --all. Historical lock entries with annotated-tag spelling such as +`v0.4.0^{}` retain their original hash verification and unchanged backup bytes. +Released v0 proto archive hashes are verified before the native materialized-snapshot +hash is calculated. Archive attributes that change proto paths or bytes block +migration. + +Internal file, directory, import-root and metadata symlinks are supported. +Logical paths keep their protobuf import names. Git targets resolve only from +the pinned tree; installed snapshots contain regular resolved bytes and work +with core.symlinks=false. Selected inputs cannot escape their owner, +cross undeclared nested repositories/submodules or form cycles. Invalid unused +auxiliary links are omitted. Migration verifies historical archive contents +separately; it never substitutes the new snapshot hash for a v0 input hash. + When reviewing v1.0, treat differences listed in this document as deliberate pre-release contract changes. A failing literal example from the earlier pilot RFC is not by itself a product defect if the current behavior matches this document and the linked `.spec` contract. diff --git a/internal/adapters/gitindex/parse.go b/internal/adapters/gitindex/parse.go new file mode 100644 index 00000000..053c7ac9 --- /dev/null +++ b/internal/adapters/gitindex/parse.go @@ -0,0 +1,56 @@ +// Package gitindex parses staged Git index records. +package gitindex + +import ( + "fmt" + "path/filepath" + "strings" +) + +// Mode is the file mode recorded in the Git index. +type Mode string + +const ( + // RegularFile is the mode of a non-executable regular file. + RegularFile Mode = "100644" + // ExecutableFile is the mode of an executable regular file. + ExecutableFile Mode = "100755" + // Symlink is the mode of a symbolic link. + Symlink Mode = "120000" + // Gitlink is the mode of a submodule entry. + Gitlink Mode = "160000" +) + +// Entry is one staged Git index record. +type Entry struct { + Mode Mode + Path string + // Raw preserves the original record without its NUL separator. + Raw string +} + +// IsRegular reports whether the entry is a regular or executable file. +func (e Entry) IsRegular() bool { + return e.Mode == RegularFile || e.Mode == ExecutableFile +} + +// Parse parses NUL-separated output from git ls-files --stage -z. +// It requires stage zero and local paths, leaving mode policy to callers. +func Parse(raw string) ([]Entry, error) { + var entries []Entry + for record := range strings.SplitSeq(raw, "\x00") { + if record == "" { + continue + } + metadata, path, ok := strings.Cut(record, "\t") + fields := strings.Fields(metadata) + if !ok || len(fields) != 3 || fields[2] != "0" { + return nil, fmt.Errorf("invalid Git index entry %q", record) + } + if !filepath.IsLocal(filepath.FromSlash(path)) { + return nil, fmt.Errorf("invalid tracked file path %q", path) + } + entries = append(entries, Entry{Mode: Mode(fields[0]), Path: path, Raw: record}) + } + return entries, nil +} diff --git a/internal/adapters/gitindex/parse_test.go b/internal/adapters/gitindex/parse_test.go new file mode 100644 index 00000000..b03cebe8 --- /dev/null +++ b/internal/adapters/gitindex/parse_test.go @@ -0,0 +1,196 @@ +package gitindex_test + +import ( + "fmt" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/easyp-tech/easyp/internal/adapters/gitindex" +) + +func TestParseModes(t *testing.T) { + t.Parallel() + tests := []struct { + name string + mode string + want gitindex.Mode + }{ + {name: "regular_file", mode: "100644", want: gitindex.RegularFile}, + {name: "executable_file", mode: "100755", want: gitindex.ExecutableFile}, + {name: "symlink", mode: "120000", want: gitindex.Symlink}, + {name: "gitlink", mode: "160000", want: gitindex.Gitlink}, + {name: "unknown_mode", mode: "100600", want: "100600"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + raw := tt.mode + " 0123456789abcdef0123456789abcdef01234567 0\tfile.proto" + + entries, err := gitindex.Parse(raw + "\x00") + + require.NoError(t, err) + assert.Equal(t, []gitindex.Entry{{Mode: tt.want, Path: "file.proto", Raw: raw}}, entries) + }) + } +} + +func TestParsePreservesPaths(t *testing.T) { + t.Parallel() + tests := []struct { + name string + path string + }{ + {name: "spaces", path: "directory name/file name.proto"}, + {name: "leading_and_trailing_spaces", path: " file.proto "}, + {name: "tabs", path: "directory\tname/file\tname.proto"}, + {name: "newlines", path: "directory\nname/file\nname.proto"}, + {name: "leading_and_trailing_whitespace", path: "\t\n file.proto \n\t"}, + {name: "non_utf8_bytes", path: "file\xff.proto"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + raw := "100644 0123456789abcdef0123456789abcdef01234567 0\t" + tt.path + + entries, err := gitindex.Parse(raw + "\x00") + + require.NoError(t, err) + assert.Equal(t, []gitindex.Entry{{Mode: gitindex.RegularFile, Path: tt.path, Raw: raw}}, entries) + }) + } +} + +func TestParseRecords(t *testing.T) { + t.Parallel() + first := "100644 0123456789abcdef0123456789abcdef01234567 0\tfirst.proto" + second := "100755 abcdef0123456789abcdef0123456789abcdef0123 0\tsecond.proto" + want := []gitindex.Entry{ + {Mode: gitindex.RegularFile, Path: "first.proto", Raw: first}, + {Mode: gitindex.ExecutableFile, Path: "second.proto", Raw: second}, + } + tests := []struct { + name string + raw string + want []gitindex.Entry + }{ + {name: "empty_output", raw: ""}, + {name: "empty_records", raw: "\x00\x00\x00"}, + {name: "multiple_records", raw: first + "\x00" + second + "\x00", want: want}, + {name: "empty_records_are_skipped", raw: "\x00" + first + "\x00\x00" + second + "\x00\x00", want: want}, + {name: "no_trailing_nul", raw: first + "\x00" + second, want: want}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + + entries, err := gitindex.Parse(tt.raw) + + require.NoError(t, err) + assert.Equal(t, tt.want, entries) + }) + } +} + +func TestParseMetadata(t *testing.T) { + t.Parallel() + tests := []struct { + name string + metadata string + }{ + {name: "opaque_object_id", metadata: "100644 opaque-object-id 0"}, + {name: "sha256_object_id", metadata: "100644 " + strings.Repeat("a", 64) + " 0"}, + {name: "metadata_whitespace", metadata: " 100644 opaque-object-id 0 "}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + raw := tt.metadata + "\tfile.proto" + + entries, err := gitindex.Parse(raw + "\x00") + + require.NoError(t, err) + assert.Equal(t, []gitindex.Entry{{Mode: gitindex.RegularFile, Path: "file.proto", Raw: raw}}, entries) + }) + } +} + +func TestParseInvalidRecords(t *testing.T) { + t.Parallel() + tests := []struct { + name string + raw string + }{ + {name: "missing_separator", raw: "100644 object-id 0 file.proto"}, + {name: "missing_metadata", raw: "\tfile.proto"}, + {name: "missing_field", raw: "100644 0\tfile.proto"}, + {name: "extra_field", raw: "100644 object-id 0 extra\tfile.proto"}, + {name: "unmerged_base_stage", raw: "100644 object-id 1\tfile.proto"}, + {name: "unmerged_ours_stage", raw: "100644 object-id 2\tfile.proto"}, + {name: "unmerged_theirs_stage", raw: "100644 object-id 3\tfile.proto"}, + {name: "invalid_stage", raw: "100644 object-id zero\tfile.proto"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + valid := "100644 object-id 0\tvalid.proto\x00" + + entries, err := gitindex.Parse(valid + tt.raw + "\x00") + + require.EqualError(t, err, fmt.Sprintf("invalid Git index entry %q", tt.raw)) + assert.Nil(t, entries) + }) + } +} + +func TestParseNonlocalPaths(t *testing.T) { + t.Parallel() + tests := []struct { + name string + path string + }{ + {name: "empty_path", path: ""}, + {name: "absolute_path", path: "/file.proto"}, + {name: "parent_directory", path: ".."}, + {name: "parent_relative_path", path: "../file.proto"}, + {name: "nested_parent_escape", path: "directory/../../file.proto"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + valid := "100644 object-id 0\tvalid.proto\x00" + raw := "100644 object-id 0\t" + tt.path + + entries, err := gitindex.Parse(valid + raw + "\x00") + + require.EqualError(t, err, fmt.Sprintf("invalid tracked file path %q", tt.path)) + assert.Nil(t, entries) + }) + } +} + +func TestEntryIsRegular(t *testing.T) { + t.Parallel() + tests := []struct { + name string + mode gitindex.Mode + want bool + }{ + {name: "regular_file", mode: "100644", want: true}, + {name: "executable_file", mode: "100755", want: true}, + {name: "symlink", mode: "120000"}, + {name: "gitlink", mode: "160000"}, + {name: "unknown_mode", mode: "100600"}, + {name: "unset_mode"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + entry := gitindex.Entry{Mode: tt.mode} + + assert.Equal(t, tt.want, entry.IsRegular()) + }) + } +} diff --git a/internal/adapters/gitmodules/checkout.go b/internal/adapters/gitmodules/checkout.go index 2bf8ce39..adcf465c 100644 --- a/internal/adapters/gitmodules/checkout.go +++ b/internal/adapters/gitmodules/checkout.go @@ -1,117 +1,10 @@ package gitmodules -import ( - "context" - "fmt" - "os" - "strings" - - moduleconfig "github.com/easyp-tech/easyp/internal/adapters/module_config" - v1 "github.com/easyp-tech/easyp/internal/config/v1" - "github.com/easyp-tech/easyp/internal/modules" -) - -// Fetch reads one revision and hashes its tracked contents. Temporary checkouts are removed before returning. -func (c *Cache) Fetch(ctx context.Context, source, version string) (modules.Fetched, error) { - cacheRoot := c.root - checkout, err := checkoutV1Module(ctx, source, version, cacheRoot) - if err != nil { - return modules.Fetched{}, fmt.Errorf("checkoutV1Module: %w", err) - } - defer func() { _ = os.RemoveAll(checkout.dir) }() - if err := moduleconfig.ValidateLegacyMajor(checkout.dir, source, version); err != nil { - return modules.Fetched{}, fmt.Errorf("ValidateLegacyMajor: %w", err) - } - files, err := trackedV1Files(ctx, checkout.dir) - if err != nil { - return modules.Fetched{}, fmt.Errorf("trackedV1Files: %w", err) - } - files = selectV1ProtoFiles(files, checkout.module.ProtoFilters) - hash, err := hashV1Files(checkout.dir, files) - if err != nil { - return modules.Fetched{}, fmt.Errorf("hashV1Files: %w", err) - } - if version == "" { - version = checkout.commit - } - module, bindings, err := c.resolveBSRDependencies(ctx, checkout.module) - if err != nil { - return modules.Fetched{}, fmt.Errorf("resolveBSRDependencies: %w", err) - } - return modules.Fetched{Module: module, Lock: v1.LockedModule{ - Source: source, Version: version, Commit: checkout.commit, Hash: hash, BSR: bindings, - }}, nil -} +import v1 "github.com/easyp-tech/easyp/internal/config/v1" type v1ModuleCheckout struct { - dir string - module v1.Module - commit string -} - -// checkoutV1Module selects a repository revision. The caller owns the returned -// directory and removes it after reading its metadata and tracked contents. -func checkoutV1Module(ctx context.Context, source, version, cacheRoot string) (checkout v1ModuleCheckout, err error) { - if err := v1.ValidateModuleVersion(source, version); err != nil { - return v1ModuleCheckout{}, fmt.Errorf("ValidateModuleVersion: %w", err) - } - if err := os.MkdirAll(cacheRoot, 0o755); err != nil { - return v1ModuleCheckout{}, fmt.Errorf("MkdirAll: %w", err) - } - if version == "" { - dir, module, commit, err := cloneHeadV1GitModule(ctx, source, cacheRoot) - if err != nil { - return v1ModuleCheckout{}, fmt.Errorf("cloneHeadV1GitModule: %w", err) - } - return v1ModuleCheckout{dir: dir, module: module, commit: strings.TrimSpace(commit)}, nil - } - - var dir string - // Ownership transfers to the caller only after checkout and metadata agree. - defer func() { - if err != nil && dir != "" { - _ = os.RemoveAll(dir) - } - }() - if v1.IsCommitRef(version) { - dir, err = clonePinnedV1GitModule(ctx, v1.LockedModule{Source: source, Commit: version}, cacheRoot) - if err != nil { - return v1ModuleCheckout{}, fmt.Errorf("clonePinnedV1GitModule: %w", err) - } - commit, err := gitV1(ctx, dir, "rev-parse", "HEAD") - if err != nil { - return v1ModuleCheckout{}, fmt.Errorf("gitV1: %w", err) - } - module, err := readCachedV1Module(dir, source) - if err != nil { - return v1ModuleCheckout{}, fmt.Errorf("readCachedV1Module: %w", err) - } - return v1ModuleCheckout{dir: dir, module: module, commit: strings.TrimSpace(commit)}, nil - } - - candidate, err := findV1GitModuleTag(ctx, source, version) - if err != nil { - return v1ModuleCheckout{}, fmt.Errorf("findV1GitModuleTag: %w", err) - } - dir, err = os.MkdirTemp(cacheRoot, "git-*") - if err != nil { - return v1ModuleCheckout{}, fmt.Errorf("MkdirTemp: %w", err) - } - tag := candidate.tag(strings.TrimSuffix(version, "+incompatible")) - if _, err := gitV1(ctx, "", "clone", "--quiet", "--depth=1", "--branch", tag, "--no-checkout", "--", candidate.remote, dir); err != nil { - return v1ModuleCheckout{}, fmt.Errorf("gitV1: %w", err) - } - commit, err := gitV1(ctx, dir, "rev-parse", "--verify", "refs/tags/"+tag+"^{commit}") - if err != nil { - return v1ModuleCheckout{}, fmt.Errorf("gitV1: %w", err) - } - commit = strings.TrimSpace(commit) - if _, err := gitV1(ctx, dir, "checkout", "--quiet", "--detach", commit); err != nil { - return v1ModuleCheckout{}, fmt.Errorf("gitV1: %w", err) - } - module, err := moduleconfig.ReadGitDependencyAt(dir, source, candidate.subdir) - if err != nil { - return v1ModuleCheckout{}, fmt.Errorf("ReadGitDependencyAt: %w", err) - } - return v1ModuleCheckout{dir: dir, module: module, commit: commit}, nil + dir string + module v1.Module + commit string + snapshot string } diff --git a/internal/adapters/gitmodules/download.go b/internal/adapters/gitmodules/download.go index 63e2f642..47d6715c 100644 --- a/internal/adapters/gitmodules/download.go +++ b/internal/adapters/gitmodules/download.go @@ -6,15 +6,13 @@ import ( "fmt" "os" "path/filepath" - "strings" moduleconfig "github.com/easyp-tech/easyp/internal/adapters/module_config" v1 "github.com/easyp-tech/easyp/internal/config/v1" - disk "github.com/easyp-tech/easyp/internal/fs/fs" ) func v1ModuleCachePath(cacheRoot string, entry v1.LockedModule) string { - return filepath.Join(cacheRoot, "modules", v1CacheSourceKey(entry.Source), entry.Commit) + return filepath.Join(cacheRoot, "modules", v1CacheSourceKey(entry.Source), v1CacheSourceKey(entry.Commit+":"+entry.Hash)) } // Install installs each locked Git commit, verifying its content hash @@ -85,74 +83,3 @@ func (c *Cache) installEntry(ctx context.Context, entry v1.LockedModule, acquire } return nil } - -func fetchPinnedV1Module(ctx context.Context, entry v1.LockedModule, cacheRoot, installed string) error { - if err := os.MkdirAll(cacheRoot, 0o755); err != nil { - return err - } - checkout, err := clonePinnedV1GitModule(ctx, entry, cacheRoot) - if err != nil { - return fmt.Errorf("clonePinnedV1GitModule: %w", err) - } - defer func() { _ = os.RemoveAll(checkout) }() - commit, err := gitV1(ctx, checkout, "rev-parse", "HEAD") - if err != nil { - return err - } - if strings.TrimSpace(commit) != entry.Commit { - return fmt.Errorf("%s: checked out commit does not match lock", entry.Source) - } - module, err := readCachedV1Module(checkout, entry.Source) - if err != nil { - return fmt.Errorf("readCachedV1Module: %w", err) - } - files, err := trackedV1Files(ctx, checkout) - if err != nil { - return fmt.Errorf("trackedV1Files: %w", err) - } - files = selectV1ProtoFiles(files, module.ProtoFilters) - actual, err := hashV1Files(checkout, files) - if err != nil { - return fmt.Errorf("hashV1Files: %w", err) - } - if actual != entry.Hash { - return fmt.Errorf("%s@%s hash mismatch: got %s, want %s; downloaded contents do not match the pinned hash; investigate repository integrity and keep protobuf.lock unchanged", entry.Source, entry.Commit, actual, entry.Hash) - } - if err := moduleconfig.ValidateLegacyMajor(checkout, entry.Source, entry.Version); err != nil { - return fmt.Errorf("ValidateLegacyMajor: %w", err) - } - parent := filepath.Dir(installed) - if err := os.MkdirAll(parent, 0o755); err != nil { - return err - } - stage, err := os.MkdirTemp(parent, "install-*") - if err != nil { - return err - } - defer func() { _ = os.RemoveAll(stage) }() - if len(module.ProtoFilters) > 0 { - // A valid Buf root may become empty after includes/excludes are applied. - for _, root := range module.Roots { - info, err := os.Lstat(filepath.Join(checkout, root)) - if err != nil { - return fmt.Errorf("Lstat: %w", err) - } - if !info.IsDir() { - return fmt.Errorf("module %s has invalid root %q: not a directory", module.Name, root) - } - if err := os.MkdirAll(filepath.Join(stage, root), 0o755); err != nil { - return fmt.Errorf("MkdirAll: %w", err) - } - } - } - for _, name := range files { - path := filepath.FromSlash(name) - if err := disk.CopyRegularFile(filepath.Join(checkout, path), filepath.Join(stage, path)); err != nil { - return fmt.Errorf("CopyRegularFile: %w", err) - } - } - if err := os.Rename(stage, installed); err != nil { - return fmt.Errorf("install %s@%s: %w", entry.Source, entry.Commit, err) - } - return nil -} diff --git a/internal/adapters/gitmodules/files.go b/internal/adapters/gitmodules/files.go index d23b9c35..8475c305 100644 --- a/internal/adapters/gitmodules/files.go +++ b/internal/adapters/gitmodules/files.go @@ -10,6 +10,7 @@ import ( "golang.org/x/mod/sumdb/dirhash" + "github.com/easyp-tech/easyp/internal/adapters/gitindex" moduleconfig "github.com/easyp-tech/easyp/internal/adapters/module_config" v1 "github.com/easyp-tech/easyp/internal/config/v1" ) @@ -17,37 +18,48 @@ import ( // trackedV1Files selects regular Git files, including those outside import roots. // Symlinks and submodules are omitted, matching Go module archive behavior. func trackedV1Files(ctx context.Context, checkout string) ([]string, error) { + entries, err := readV1GitIndex(ctx, checkout) + if err != nil { + return nil, fmt.Errorf("readV1GitIndex: %w", err) + } + files, err := regularTrackedV1Files(checkout, entries) + if err != nil { + return nil, fmt.Errorf("regularTrackedV1Files: %w", err) + } + return files, nil +} + +func readV1GitIndex(ctx context.Context, checkout string) ([]gitindex.Entry, error) { raw, err := gitV1(ctx, checkout, "ls-files", "--stage", "-z") if err != nil { return nil, fmt.Errorf("gitV1: %w", err) } + entries, err := gitindex.Parse(raw) + if err != nil { + return nil, fmt.Errorf("Parse: %w", err) + } + return entries, nil +} + +// Snapshot hashes and installation use the same regular-file selection. +// Policy checks use Git modes even when symlinks are materialized as text. +func regularTrackedV1Files(checkout string, entries []gitindex.Entry) ([]string, error) { var files []string - for _, entry := range strings.Split(raw, "\x00") { - if entry == "" { - continue - } - metadata, name, ok := strings.Cut(entry, "\t") - fields := strings.Fields(metadata) - if !ok || len(fields) != 3 || fields[2] != "0" { - return nil, fmt.Errorf("invalid Git index entry %q", entry) - } - if !filepath.IsLocal(filepath.FromSlash(name)) { - return nil, fmt.Errorf("invalid tracked file path %q", name) - } - switch fields[0] { - case "120000", "160000": - if fields[0] == "120000" && moduleconfig.IsGitDependencyConfigFile(filepath.Base(name)) { - return nil, fmt.Errorf("non-regular dependency config %q in Git index", name) + for _, entry := range entries { + switch entry.Mode { + case gitindex.Symlink, gitindex.Gitlink: + if entry.Mode == gitindex.Symlink && moduleconfig.IsGitDependencyConfigFile(filepath.Base(entry.Path)) { + return nil, fmt.Errorf("non-regular dependency config %q in Git index", entry.Path) } continue - case "100644", "100755": + case gitindex.RegularFile, gitindex.ExecutableFile: default: - return nil, fmt.Errorf("unsupported Git file mode %q for %q", fields[0], name) + return nil, fmt.Errorf("unsupported Git file mode %q for %q", entry.Mode, entry.Path) } - if _, err := regularV1File(filepath.Join(checkout, filepath.FromSlash(name))); err != nil { + if _, err := regularV1File(filepath.Join(checkout, filepath.FromSlash(entry.Path))); err != nil { return nil, fmt.Errorf("regularV1File: %w", err) } - files = append(files, name) + files = append(files, entry.Path) } return files, nil } diff --git a/internal/adapters/gitmodules/git_source.go b/internal/adapters/gitmodules/git_source.go index 5bea54eb..b1366b7a 100644 --- a/internal/adapters/gitmodules/git_source.go +++ b/internal/adapters/gitmodules/git_source.go @@ -11,7 +11,6 @@ import ( "golang.org/x/mod/semver" - moduleconfig "github.com/easyp-tech/easyp/internal/adapters/module_config" v1 "github.com/easyp-tech/easyp/internal/config/v1" ) @@ -224,107 +223,3 @@ func listV1CandidateTags(ctx context.Context, candidate v1GitModuleCandidate) ([ slices.Sort(versions) return versions, nil } - -// cloneHeadV1GitModule finds the repository default branch containing source. -// The caller owns the returned checkout and must remove it. -func cloneHeadV1GitModule(ctx context.Context, source, cacheRoot string) (string, v1.Module, string, error) { - candidates, err := v1GitModuleCandidates(source) - if err != nil { - return "", v1.Module{}, "", err - } - var firstErr, moduleErr error - for _, candidate := range candidates { - if err := ctx.Err(); err != nil { - return "", v1.Module{}, "", err - } - checkout, err := os.MkdirTemp(cacheRoot, "git-*") - if err != nil { - return "", v1.Module{}, "", err - } - module, commit, cloned, err := checkoutHeadV1GitModuleCandidate(ctx, checkout, source, candidate) - if err != nil { - if firstErr == nil { - firstErr = err - } - if cloned { - moduleErr = err - } - if removeErr := os.RemoveAll(checkout); removeErr != nil { - return "", v1.Module{}, "", removeErr - } - continue - } - return checkout, module, commit, nil - } - if moduleErr != nil { - return "", v1.Module{}, "", fmt.Errorf("module %s was not found at a Git repository HEAD: %w", source, moduleErr) - } - if firstErr != nil { - return "", v1.Module{}, "", fmt.Errorf("git repository for %s was not found: %w", source, firstErr) - } - return "", v1.Module{}, "", fmt.Errorf("no Git repository candidate for %s", source) -} - -func checkoutHeadV1GitModuleCandidate(ctx context.Context, checkout, source string, candidate v1GitModuleCandidate) (v1.Module, string, bool, error) { - if _, err := gitV1(ctx, "", "clone", "--quiet", "--depth=1", "--no-checkout", "--", candidate.remote, checkout); err != nil { - return v1.Module{}, "", false, err - } - commit, err := gitV1(ctx, checkout, "rev-parse", "HEAD") - if err != nil { - return v1.Module{}, "", true, err - } - commit = strings.TrimSpace(commit) - if _, err := gitV1(ctx, checkout, "checkout", "--quiet", "--detach", commit); err != nil { - return v1.Module{}, "", true, err - } - module, err := moduleconfig.ReadGitDependencyAt(checkout, source, candidate.subdir) - if err != nil { - return v1.Module{}, "", true, err - } - return module, commit, true, nil -} - -// clonePinnedV1GitModule finds a repository containing the locked commit and -// the requested module. It does not depend on the tag still pointing there. -func clonePinnedV1GitModule(ctx context.Context, entry v1.LockedModule, cacheRoot string) (string, error) { - candidates, err := v1GitModuleCandidates(entry.Source) - if err != nil { - return "", fmt.Errorf("v1GitModuleCandidates: %w", err) - } - var firstErr error - var moduleErr error - for _, candidate := range candidates { - if err := ctx.Err(); err != nil { - return "", err - } - checkout, err := os.MkdirTemp(cacheRoot, "git-*") - if err != nil { - return "", fmt.Errorf("MkdirTemp: %w", err) - } - cloned, err := checkoutPinnedV1GitModuleCandidate(ctx, checkout, entry, candidate) - if err != nil { - if cloned { - moduleErr = err - } - if firstErr == nil { - firstErr = err - } - if removeErr := os.RemoveAll(checkout); removeErr != nil { - return "", fmt.Errorf("RemoveAll: %w", removeErr) - } - continue - } - return checkout, nil - } - if moduleErr != nil { - return "", fmt.Errorf("%s: %w", entry.Source, moduleErr) - } - if firstErr == nil { - return "", fmt.Errorf("%s: no Git repository candidate", entry.Source) - } - return "", fmt.Errorf("%s: could not fetch locked commit %s; check repository access before changing the lock: %w", entry.Source, entry.Commit, firstErr) -} - -func checkoutPinnedV1GitModuleCandidate(ctx context.Context, checkout string, entry v1.LockedModule, candidate v1GitModuleCandidate) (bool, error) { - return checkoutCachedCommit(ctx, checkout, entry, candidate) -} diff --git a/internal/adapters/gitmodules/migration.go b/internal/adapters/gitmodules/migration.go index 679336b0..ed2aaec5 100644 --- a/internal/adapters/gitmodules/migration.go +++ b/internal/adapters/gitmodules/migration.go @@ -4,15 +4,9 @@ import ( "context" "errors" "fmt" - "io" "os" - "path" - "path/filepath" - "slices" - "strings" "golang.org/x/mod/semver" - "golang.org/x/mod/sumdb/dirhash" moduleconfig "github.com/easyp-tech/easyp/internal/adapters/module_config" v1 "github.com/easyp-tech/easyp/internal/config/v1" @@ -20,7 +14,7 @@ import ( ) // FetchMigration verifies a legacy installed-tree hash before calculating the -// v1 tracked-checkout hash. Legacy repositories must preserve their proto source +// v1 logical snapshot hash. Legacy repositories must preserve their proto source // selection and import names. An empty version requires an empty legacyHash and // permits initial resolution; native v1 repositories need no legacy comparison. func (c *Cache) FetchMigration(ctx context.Context, source, version, legacyHash string) (fetched modules.Fetched, err error) { @@ -32,42 +26,44 @@ func (c *Cache) FetchMigration(ctx context.Context, source, version, legacyHash return modules.Fetched{}, fmt.Errorf("checkoutV1Module: %w", err) } defer func() { - removeErr := os.RemoveAll(checkout.dir) - if removeErr != nil { - fetched = modules.Fetched{} - err = errors.Join(err, fmt.Errorf("RemoveAll: %w", removeErr)) + for _, directory := range []string{checkout.snapshot, checkout.dir} { + removeErr := os.RemoveAll(directory) + if removeErr != nil { + fetched = modules.Fetched{} + err = errors.Join(err, fmt.Errorf("RemoveAll: %w", removeErr)) + } } }() - if err := moduleconfig.ValidateLegacyMajor(checkout.dir, source, version); err != nil { + if err := moduleconfig.ValidateLegacyMajor(checkout.snapshot, source, version); err != nil { return modules.Fetched{}, fmt.Errorf("ValidateLegacyMajor: %w", err) } - files, err := migrationTrackedFiles(ctx, checkout.dir) + tracked, err := migrationTrackedFiles(ctx, checkout.dir) if err != nil { return modules.Fetched{}, fmt.Errorf("migrationTrackedFiles: %w", err) } - if err := validateMigrationLegacyReplacements(checkout.dir, files); err != nil { + files, err := snapshotV1Files(checkout.snapshot) + if err != nil { + return modules.Fetched{}, fmt.Errorf("snapshotV1Files: %w", err) + } + if err := validateMigrationLegacyReplacements(checkout.snapshot, files); err != nil { return modules.Fetched{}, fmt.Errorf("validateMigrationLegacyReplacements: %w", err) } - native, err := hasNativeMigrationModule(checkout.dir, files, source) + native, err := hasNativeMigrationModule(checkout.snapshot, files, source) if err != nil { return modules.Fetched{}, fmt.Errorf("hasNativeMigrationModule: %w", err) } + _, err = migrationSelectedFiles(checkout.snapshot, checkout.module) + if err != nil { + return modules.Fetched{}, fmt.Errorf("migrationSelectedFiles: %w", err) + } if !native || legacyHash != "" { - oldHash, err := hashMigrationLegacyFiles(checkout.dir, files) - if err != nil { - return modules.Fetched{}, fmt.Errorf("hashMigrationLegacyFiles: %w", err) - } - if legacyHash != "" && oldHash != legacyHash { - return modules.Fetched{}, fmt.Errorf("legacy hash mismatch for %s at %s: got %s, want %s", source, checkout.commit, oldHash, legacyHash) - } - if err := validateMigrationSelection(checkout.dir, files, checkout.module); err != nil { - return modules.Fetched{}, fmt.Errorf("validateMigrationSelection: %w", err) + if err := verifyMigrationLegacyHash(ctx, checkout, source, legacyHash, tracked); err != nil { + return modules.Fetched{}, fmt.Errorf("verifyMigrationLegacyHash: %w", err) } } - files = selectV1ProtoFiles(files, checkout.module.ProtoFilters) - hash, err := hashV1Files(checkout.dir, files) + hash, err := hashSnapshotV1Files(checkout.snapshot) if err != nil { - return modules.Fetched{}, fmt.Errorf("hashV1Files: %w", err) + return modules.Fetched{}, fmt.Errorf("hashSnapshotV1Files: %w", err) } if version == "" { version = checkout.commit @@ -80,97 +76,3 @@ func (c *Cache) FetchMigration(ctx context.Context, source, version, legacyHash Source: source, Version: version, Commit: checkout.commit, Hash: hash, BSR: bindings, }}, nil } - -func migrationTrackedFiles(ctx context.Context, checkout string) ([]string, error) { - files, err := trackedV1Files(ctx, checkout) - if err != nil { - return nil, fmt.Errorf("trackedV1Files: %w", err) - } - // A Git symlink may be checked out as a regular file on systems that do - // not support symlinks. Inspect Git modes as well as filesystem modes. - raw, err := gitV1(ctx, checkout, "ls-files", "--stage", "-z") - if err != nil { - return nil, fmt.Errorf("gitV1: %w", err) - } - for _, entry := range strings.Split(strings.TrimSuffix(raw, "\x00"), "\x00") { - if entry == "" { - continue - } - if !strings.HasPrefix(entry, "100644 ") && !strings.HasPrefix(entry, "100755 ") { - return nil, fmt.Errorf("unsupported non-regular Git mode in %q", entry) - } - } - return files, nil -} - -// hashMigrationLegacyFiles reproduces the old installer's path rewrite without -// creating an installed tree. All tracked files participate, including files -// outside import roots and non-proto files. Directory nodes also participate in -// collision detection because buildInstallTree created them before hashing. -func hashMigrationLegacyFiles(checkout string, files []string) (string, error) { - directories := make(map[string]bool) - for _, name := range files { - if !filepath.IsLocal(filepath.FromSlash(name)) || path.Clean(name) != name || strings.Contains(name, "\\") { - return "", fmt.Errorf("unsupported tracked path %q", name) - } - if _, err := regularV1File(filepath.Join(checkout, filepath.FromSlash(name))); err != nil { - return "", fmt.Errorf("regularV1File: %w", err) - } - for directory := path.Dir(name); directory != "."; directory = path.Dir(directory) { - directories[directory] = true - } - } - roots, err := readMigrationLegacyRoots(checkout, files) - if err != nil { - return "", fmt.Errorf("readMigrationLegacyRoots: %w", err) - } - - installedDirectories := make(map[string]bool) - for directory := range directories { - for destination := renameMigrationLegacyFile(directory, roots); destination != "."; destination = path.Dir(destination) { - installedDirectories[destination] = true - } - } - originals := make(map[string]string, len(files)) - names := make([]string, 0, len(files)) - for _, original := range files { - name := renameMigrationLegacyFile(original, roots) - if other, exists := originals[name]; exists { - return "", fmt.Errorf("legacy file collision at %q between %q and %q", name, other, original) - } - originals[name] = original - names = append(names, name) - for directory := path.Dir(name); directory != "."; directory = path.Dir(directory) { - installedDirectories[directory] = true - } - } - slices.Sort(names) - for _, name := range names { - if installedDirectories[name] { - return "", fmt.Errorf("legacy directory/file collision at %q from %q", name, originals[name]) - } - } - hash, err := dirhash.Hash1(names, func(name string) (io.ReadCloser, error) { - file, err := os.Open(filepath.Join(checkout, filepath.FromSlash(originals[name]))) - if err != nil { - return nil, fmt.Errorf("Open: %w", err) - } - return file, nil - }) - if err != nil { - return "", fmt.Errorf("Hash1: %w", err) - } - return hash, nil -} - -// The old renamer selected the first matching prefix, without sorting, -// cleaning, or repeatedly stripping roots from the resulting path. -func renameMigrationLegacyFile(name string, roots []string) string { - for _, root := range roots { - prefix := root + "/" - if strings.HasPrefix(name, prefix) { - return strings.TrimPrefix(name, prefix) - } - } - return name -} diff --git a/internal/adapters/gitmodules/migration_archive.go b/internal/adapters/gitmodules/migration_archive.go new file mode 100644 index 00000000..60b8cd42 --- /dev/null +++ b/internal/adapters/gitmodules/migration_archive.go @@ -0,0 +1,336 @@ +package gitmodules + +import ( + "archive/zip" + "bytes" + "context" + "errors" + "fmt" + "io" + "io/fs" + "os" + "path" + "path/filepath" + "slices" + "strings" + "time" + + "golang.org/x/mod/sumdb/dirhash" + + "github.com/easyp-tech/easyp/internal/sourceview" +) + +// hashMigrationProtoArchive preserves the regular-file proof helper. Production +// migration uses both evidenced installer policies through migrationArchiveHashes. +func hashMigrationProtoArchive(ctx context.Context, checkout string, files []string) (string, error) { + roots, err := readMigrationLegacyRoots(checkout, files) + if err != nil { + return "", fmt.Errorf("readMigrationLegacyRoots: %w", err) + } + selected := make(map[string][]byte) + for _, name := range files { + if path.Ext(name) != ".proto" { + continue + } + data, err := os.ReadFile(filepath.Join(checkout, filepath.FromSlash(name))) + if err != nil { + return "", fmt.Errorf("ReadFile: %w", err) + } + selected[renameMigrationLegacyFile(name, roots)] = data + } + nodes, err := readMigrationProtoArchive(ctx, checkout, "HEAD", files) + if err != nil { + return "", fmt.Errorf("readMigrationProtoArchive: %w", err) + } + hashes, err := migrationArchiveHashes(ctx, nodes, roots, selected) + if err != nil { + return "", fmt.Errorf("migrationArchiveHashes: %w", err) + } + return hashes[0], nil +} + +// readMigrationProtoArchive reads only committed ZIP entries. Link payloads are +// pointer strings; they are never extracted onto, or resolved against, the host. +// Git reproduces the released *.proto pathspec and export-ignore/export-subst. +func readMigrationProtoArchive(ctx context.Context, checkout, commit string, files []string) (nodes []migrationArchiveNode, resultErr error) { + directory, err := os.MkdirTemp("", "easyp-migration-archive-") + if err != nil { + return nil, fmt.Errorf("MkdirTemp: %w", err) + } + defer func() { + removeErr := os.RemoveAll(directory) + if removeErr != nil { + resultErr = errors.Join(resultErr, fmt.Errorf("RemoveAll: %w", removeErr)) + } + }() + archivePath := filepath.Join(directory, "protos.zip") + _, err = gitV1(ctx, checkout, "archive", "--format=zip", "--output="+archivePath, commit, "--", "*.proto") + if err != nil { + return nil, fmt.Errorf("gitV1: %w", err) + } + archive, err := zip.OpenReader(archivePath) + if err != nil { + return nil, fmt.Errorf("OpenReader: %w", err) + } + defer func() { + closeErr := archive.Close() + if closeErr != nil { + resultErr = errors.Join(resultErr, fmt.Errorf("Close: %w", closeErr)) + } + }() + tracked := make(map[string]bool, len(files)) + for _, name := range files { + tracked[name] = true + } + for _, file := range archive.File { + err = ctx.Err() + if err != nil { + return nil, fmt.Errorf("Err: %w", err) + } + name := strings.TrimSuffix(file.Name, "/") + if !fs.ValidPath(name) || path.Clean(name) != name || strings.ContainsAny(name, "\\\x00") { + return nil, fmt.Errorf("unsupported archive path %q", file.Name) + } + mode := file.Mode() + if !mode.IsDir() && !mode.IsRegular() && mode&fs.ModeSymlink == 0 { + return nil, fmt.Errorf("unsupported non-regular archive file %q", name) + } + if !mode.IsDir() && !tracked[name] { + return nil, fmt.Errorf("unsupported untracked archive file %q", name) + } + var data []byte + if !mode.IsDir() { + data, err = readMigrationArchiveFile(file) + if err != nil { + return nil, fmt.Errorf("readMigrationArchiveFile: %w", err) + } + } + nodes = append(nodes, migrationArchiveNode{name: file.Name, mode: mode, data: data}) + } + return nodes, nil +} + +// migrationArchiveHashes independently reconstructs released installer layouts: +// v0.15 renamed node paths and retained pointer strings; v0.16/v0.17 additionally +// renamed each pointer's lexically resolved target. A successful candidate must +// preserve the complete proto import namespace and bytes of the current view. +func migrationArchiveHashes(ctx context.Context, nodes []migrationArchiveNode, roots []string, selected map[string][]byte) ([]string, error) { + var hashes []string + var failures []error + for _, rewrite := range []bool{false, true} { + installed, names, err := installMigrationArchive(nodes, roots, rewrite) + if err != nil { + failures = append(failures, fmt.Errorf("installMigrationArchive: %w", err)) + continue + } + view := sourceview.New(installed) + legacy := make(map[string][]byte) + for _, name := range names { + file, openErr := view.Open(ctx, name) + if openErr != nil { + err = fmt.Errorf("Open: %w", openErr) + break + } + data, readErr := io.ReadAll(file) + closeErr := file.Close() + err = errors.Join(readErr, closeErr) + if err != nil { + err = fmt.Errorf("ReadAll: %w", err) + break + } + if path.Ext(name) == ".proto" { + legacy[name] = data + } + } + if err == nil { + err = validateMigrationSourceContents("legacy archive", legacy, selected) + } + if err != nil { + failures = append(failures, err) + continue + } + hash, err := dirhash.Hash1(names, func(name string) (io.ReadCloser, error) { + file, err := view.Open(ctx, name) + if err != nil { + return nil, fmt.Errorf("Open: %w", err) + } + return file, nil + }) + if err != nil { + failures = append(failures, fmt.Errorf("Hash1: %w", err)) + continue + } + if !slices.Contains(hashes, hash) { + hashes = append(hashes, hash) + } + } + if len(hashes) == 0 { + return nil, errors.Join(failures...) + } + return hashes, nil +} + +func installMigrationArchive(nodes []migrationArchiveNode, roots []string, rewrite bool) (migrationArchiveFS, []string, error) { + installed := migrationArchiveFS{".": {name: ".", mode: fs.ModeDir}} + originals := make(map[string]string) + directories := make(map[string]bool) + for _, original := range nodes { + // Retain trailing slashes while renaming directory nodes, as extract.Archive did. + destination := path.Clean(renameMigrationLegacyFile(original.name, roots)) + if original.mode.IsDir() { + for directory := destination; directory != "."; directory = path.Dir(directory) { + directories[directory] = true + } + continue + } + if other, exists := originals[destination]; exists { + return nil, nil, fmt.Errorf("legacy file collision at %q between %q and %q", destination, other, original.name) + } + node := original + node.name = destination + if rewrite && node.mode&fs.ModeSymlink != 0 { + target, err := rewriteMigrationArchiveTarget(original.name, string(node.data), destination, roots) + if err != nil { + return nil, nil, fmt.Errorf("rewriteMigrationArchiveTarget: %w", err) + } + node.data = []byte(target) + } + installed[destination], originals[destination] = node, original.name + for directory := path.Dir(destination); directory != "."; directory = path.Dir(directory) { + directories[directory] = true + } + } + names := make([]string, 0, len(originals)) + for name := range originals { + if directories[name] { + return nil, nil, fmt.Errorf("legacy directory/file collision at %q", name) + } + names = append(names, name) + } + for directory := range directories { + installed[directory] = migrationArchiveNode{name: directory, mode: fs.ModeDir} + } + slices.Sort(names) + return installed, names, nil +} + +func rewriteMigrationArchiveTarget(original, target, destination string, roots []string) (string, error) { + if strings.ContainsAny(target, "\\\x00") || strings.HasPrefix(target, "//") || (len(target) >= 2 && target[1] == ':') { + return "", sourceview.ErrUnsupported + } + if path.IsAbs(target) { + return "", sourceview.ErrOutsideRoot + } + resolved := path.Join(path.Dir(original), target) + if !fs.ValidPath(resolved) { + return "", sourceview.ErrOutsideRoot + } + renamed := path.Clean(renameMigrationLegacyFile(resolved, roots)) + relative, err := filepath.Rel(filepath.FromSlash(path.Dir(destination)), filepath.FromSlash(renamed)) + if err != nil { + return "", fmt.Errorf("Rel: %w", err) + } + return filepath.ToSlash(relative), nil +} + +// validateMigrationArchiveRegularSources guards archive attributes during new +// acquisition without requiring an old installer to support a logical alias. +func validateMigrationArchiveRegularSources(nodes []migrationArchiveNode, snapshot string, files []string) error { + archived := make(map[string]migrationArchiveNode, len(nodes)) + for _, node := range nodes { + archived[node.name] = node + } + for _, name := range files { + if path.Ext(name) != ".proto" { + continue + } + node, exists := archived[name] + if !exists { + return fmt.Errorf("legacy archive source selection omits %q; manual migration is required", name) + } + current, err := os.ReadFile(filepath.Join(snapshot, filepath.FromSlash(name))) + if err != nil { + return fmt.Errorf("ReadFile: %w", err) + } + if !bytes.Equal(node.data, current) { + return fmt.Errorf("legacy archive source selection changes %q contents; manual migration is required", name) + } + } + return nil +} + +func readMigrationArchiveFile(file *zip.File) (content []byte, resultErr error) { + reader, err := file.Open() + if err != nil { + return nil, fmt.Errorf("Open: %w", err) + } + defer func() { + closeErr := reader.Close() + if closeErr != nil { + resultErr = errors.Join(resultErr, fmt.Errorf("Close: %w", closeErr)) + } + }() + content, err = io.ReadAll(reader) + if err != nil { + return nil, fmt.Errorf("ReadAll: %w", err) + } + return content, nil +} + +// migrationArchiveFS contains only immutable installed ZIP nodes. Open never +// follows links; sourceview is the only resolver and cannot leave this map. +type migrationArchiveFS map[string]migrationArchiveNode + +func (f migrationArchiveFS) Lstat(name string) (fs.FileInfo, error) { + if !fs.ValidPath(name) { + return nil, &fs.PathError{Op: "Lstat", Path: name, Err: fs.ErrInvalid} + } + node, exists := f[name] + if !exists { + return nil, &fs.PathError{Op: "Lstat", Path: name, Err: fs.ErrNotExist} + } + return node, nil +} + +func (f migrationArchiveFS) ReadLink(name string) (string, error) { + info, err := f.Lstat(name) + if err != nil { + return "", fmt.Errorf("Lstat: %w", err) + } + if info.Mode()&fs.ModeSymlink == 0 { + return "", &fs.PathError{Op: "ReadLink", Path: name, Err: fs.ErrInvalid} + } + return string(f[name].data), nil +} + +func (f migrationArchiveFS) Open(name string) (fs.File, error) { + info, err := f.Lstat(name) + if err != nil { + return nil, fmt.Errorf("Lstat: %w", err) + } + if !info.Mode().IsRegular() { + return nil, &fs.PathError{Op: "Open", Path: name, Err: sourceview.ErrUnsupported} + } + return migrationArchiveFile{Reader: bytes.NewReader(f[name].data), info: info}, nil +} + +type migrationArchiveNode struct { + name string + mode fs.FileMode + data []byte +} + +func (n migrationArchiveNode) Name() string { return path.Base(n.name) } +func (n migrationArchiveNode) Size() int64 { return int64(len(n.data)) } +func (n migrationArchiveNode) Mode() fs.FileMode { return n.mode } +func (n migrationArchiveNode) ModTime() time.Time { return time.Time{} } +func (n migrationArchiveNode) IsDir() bool { return n.mode.IsDir() } +func (n migrationArchiveNode) Sys() any { return nil } + +type migrationArchiveFile struct { + *bytes.Reader + info fs.FileInfo +} + +func (f migrationArchiveFile) Stat() (fs.FileInfo, error) { return f.info, nil } +func (migrationArchiveFile) Close() error { return nil } diff --git a/internal/adapters/gitmodules/migration_archive_alias_test.go b/internal/adapters/gitmodules/migration_archive_alias_test.go new file mode 100644 index 00000000..f0cc2fbc --- /dev/null +++ b/internal/adapters/gitmodules/migration_archive_alias_test.go @@ -0,0 +1,209 @@ +package gitmodules + +import ( + "io/fs" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/easyp-tech/easyp/internal/sourceview" +) + +func TestFetchMigrationVerifiesHistoricalArchivedFileAliases(t *testing.T) { + t.Parallel() + tests := []struct { + name string + config string + files map[string]string + link string + target string + installed map[string]string + }{ + { + name: "default roots preserve file pointer", files: map[string]string{"target.proto": "syntax = \"proto3\";"}, + link: "alias.proto", target: "target.proto", installed: map[string]string{"target.proto": "syntax = \"proto3\";", "alias.proto": "syntax = \"proto3\";"}, + }, + { + name: "stripped roots preserve relative file pointer", config: "generate:\n inputs: [{directory: {path: proto, root: proto}}]\n", + files: map[string]string{"proto/target.proto": "syntax = \"proto3\";"}, link: "proto/alias.proto", target: "target.proto", + installed: map[string]string{"target.proto": "syntax = \"proto3\";", "alias.proto": "syntax = \"proto3\";"}, + }, + { + name: "later installer rewrites cross root pointer", config: "generate:\n inputs: [{directory: {path: proto, root: proto}}, {directory: {path: shared, root: shared}}]\n", + files: map[string]string{"shared/target.proto": "syntax = \"proto3\";"}, link: "proto/alias.proto", target: "../shared/target.proto", + installed: map[string]string{"target.proto": "syntax = \"proto3\";", "alias.proto": "syntax = \"proto3\";"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + if tt.config != "" { + tt.files["easyp.yaml"] = tt.config + } + repository, _ := migrationTestRepository(t, tt.files) + commit := migrationTestCommitSymlink(t, repository, tt.link, tt.target) + oldHash := migrationTestHash(t, tt.installed) + cacheDirectory := t.TempDir() + fetched, err := (&Cache{root: cacheDirectory}).FetchMigration(t.Context(), repository, commit, oldHash) + require.NoError(t, err) + assert.Equal(t, commit, fetched.Lock.Commit) + assert.Equal(t, commit, fetched.Lock.Version) + assert.True(t, strings.HasPrefix(fetched.Lock.Hash, "h1:")) + ordinary, err := (&Cache{root: t.TempDir()}).Fetch(t.Context(), repository, commit) + require.NoError(t, err) + assert.Equal(t, ordinary.Lock.Hash, fetched.Lock.Hash) + assert.Equal(t, oldHash, migrationTestHash(t, tt.installed), "the recorded historical proof remains unchanged") + migrationTestAssertNoCheckout(t, cacheDirectory) + }) + } +} + +func TestFetchMigrationInitialRootAliasDoesNotInventAnArchivedNamespace(t *testing.T) { + t.Parallel() + files := map[string]string{ + "easyp.yaml": "generate:\n inputs: [{directory: {path: proto, root: proto}}]\n", + "real/file.proto": "syntax = \"proto3\";", + } + repository, _ := migrationTestRepository(t, files) + commit := migrationTestCommitSymlink(t, repository, "proto", "real") + cacheDirectory := t.TempDir() + fetched, err := (&Cache{root: cacheDirectory}).FetchMigration(t.Context(), repository, "", "") + require.NoError(t, err) + assert.Equal(t, []string{"proto"}, fetched.Module.Roots) + assert.Equal(t, commit, fetched.Lock.Commit) + migrationTestAssertNoCheckout(t, cacheDirectory) + // A real old archive retained real/file.proto: it never contained the + // proto directory alias or the new file.proto import namespace. + legacyHash := migrationTestHash(t, map[string]string{"real/file.proto": files["real/file.proto"]}) + _, err = (&Cache{root: cacheDirectory}).FetchMigration(t.Context(), repository, commit, legacyHash) + require.ErrorContains(t, err, "source selection") + require.ErrorContains(t, err, "manual migration") + migrationTestAssertNoCheckout(t, cacheDirectory) +} + +func TestFetchMigrationRelevantAliasStillRequiresCompleteSourceOwnership(t *testing.T) { + t.Parallel() + for _, link := range []string{"root", "metadata", "file"} { + t.Run(link, func(t *testing.T) { + t.Parallel() + files := map[string]string{ + "proto/file.proto": "syntax = \"proto3\";", + "extras/extra.proto": "unrelated source scope", + "easyp.yaml": "generate:\n inputs: [{directory: {path: proto, root: proto}}]\n", + } + name, target := "proto/alias.proto", "file.proto" + if link == "root" { + files["easyp.yaml"] = "generate:\n inputs: [{directory: {path: api, root: api}}]\n" + name, target = "api", "proto" + } + if link == "metadata" { + files["config.yaml"] = files["easyp.yaml"] + delete(files, "easyp.yaml") + name, target = "easyp.yaml", "config.yaml" + } + repository, _ := migrationTestRepository(t, files) + migrationTestCommitSymlink(t, repository, name, target) + cacheDirectory := t.TempDir() + _, err := (&Cache{root: cacheDirectory}).FetchMigration(t.Context(), repository, "", "") + require.ErrorContains(t, err, "source selection") + require.ErrorContains(t, err, "extras/extra.proto") + migrationTestAssertNoCheckout(t, cacheDirectory) + }) + } +} + +func TestMigrationArchiveCandidatesPreserveBothEvidencedPointerPolicies(t *testing.T) { + t.Parallel() + repository, _ := migrationTestRepository(t, map[string]string{"shared/target.proto": "proto"}) + commit := migrationTestCommitSymlink(t, repository, "proto/alias.proto", "../shared/target.proto") + tracked, err := migrationTrackedFiles(t.Context(), repository) + require.NoError(t, err) + nodes, err := readMigrationProtoArchive(t.Context(), repository, commit, tracked.trackedFiles) + require.NoError(t, err) + roots := []string{"proto", "shared"} + v15, _, err := installMigrationArchive(nodes, roots, false) + require.NoError(t, err) + assert.Equal(t, "../shared/target.proto", string(v15["alias.proto"].data)) + _, err = sourceview.New(v15).Open(t.Context(), "alias.proto") + require.ErrorIs(t, err, sourceview.ErrOutsideRoot) + v16, _, err := installMigrationArchive(nodes, roots, true) + require.NoError(t, err) + assert.Equal(t, "target.proto", string(v16["alias.proto"].data)) + selected := map[string][]byte{"alias.proto": []byte("proto"), "target.proto": []byte("proto")} + hashes, err := migrationArchiveHashes(t.Context(), nodes, roots, selected) + require.NoError(t, err) + assert.Equal(t, []string{migrationTestHash(t, map[string]string{"alias.proto": "proto", "target.proto": "proto"})}, hashes) +} + +func TestMigrationArchiveProofRejectsInvalidPointerLeaves(t *testing.T) { + t.Parallel() + tests := []struct { + name string + target string + want error + }{ + {name: "external", target: "/outside/target.proto", want: sourceview.ErrOutsideRoot}, + {name: "drive", target: "C:/outside", want: sourceview.ErrUnsupported}, + {name: "NUL", target: "bad\x00target", want: sourceview.ErrUnsupported}, + {name: "dangling", target: "missing.proto", want: fs.ErrNotExist}, + {name: "cycle", target: "alias.proto", want: sourceview.ErrCycle}, + {name: "directory leaf", target: "directory", want: sourceview.ErrUnsupported}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + nodes := []migrationArchiveNode{ + {name: "directory/", mode: fs.ModeDir}, + {name: "directory/target.proto", data: []byte("proto")}, + {name: "alias.proto", mode: fs.ModeSymlink, data: []byte(tt.target)}, + } + _, err := migrationArchiveHashes(t.Context(), nodes, nil, map[string][]byte{"alias.proto": []byte("proto"), "directory/target.proto": []byte("proto")}) + require.ErrorIs(t, err, tt.want) + }) + } +} + +func TestFetchMigrationNativeInitialAliasesStillRejectImportCollisions(t *testing.T) { + t.Parallel() + repository, _ := migrationTestRepository(t, map[string]string{"real/file.proto": "proto"}) + migrationTestWriteFiles(t, repository, map[string]string{"protobuf.mod": "module " + repository + "\nroots one two\n"}) + migrationTestCommitSymlink(t, repository, "one", "real") + migrationTestCommitSymlink(t, repository, "two", "real") + cacheDirectory := t.TempDir() + _, err := (&Cache{root: cacheDirectory}).FetchMigration(t.Context(), repository, "", "") + require.ErrorContains(t, err, "source selection collides") + require.ErrorContains(t, err, "file.proto") + migrationTestAssertNoCheckout(t, cacheDirectory) +} + +func TestFetchMigrationArchiveCannotBorrowOmittedAliasTarget(t *testing.T) { + t.Parallel() + files := map[string]string{"target.txt": "unarchived target"} + repository, _ := migrationTestRepository(t, files) + commit := migrationTestCommitSymlink(t, repository, "alias.proto", "target.txt") + legacyHash := migrationTestHash(t, map[string]string{"alias.proto": files["target.txt"]}) + cacheDirectory := t.TempDir() + _, err := (&Cache{root: cacheDirectory}).FetchMigration(t.Context(), repository, commit, legacyHash) + require.ErrorIs(t, err, fs.ErrNotExist) + assert.ErrorContains(t, err, "migrationArchiveHashes") + migrationTestAssertNoCheckout(t, cacheDirectory) +} + +func TestFetchMigrationArchiveAliasReadsArchivedTargetBytes(t *testing.T) { + t.Parallel() + files := map[string]string{ + ".gitattributes": "target.proto export-subst\n", + "target.proto": "// $Format:%H$\nsyntax = \"proto3\";", + } + repository, _ := migrationTestRepository(t, files) + commit := migrationTestCommitSymlink(t, repository, "alias.proto", "target.proto") + archived := strings.ReplaceAll(files["target.proto"], "$Format:%H$", commit) + legacyHash := migrationTestHash(t, map[string]string{"alias.proto": archived, "target.proto": archived}) + cacheDirectory := t.TempDir() + _, err := (&Cache{root: cacheDirectory}).FetchMigration(t.Context(), repository, commit, legacyHash) + require.ErrorContains(t, err, "source selection") + require.ErrorContains(t, err, "manual migration") + migrationTestAssertNoCheckout(t, cacheDirectory) +} diff --git a/internal/adapters/gitmodules/migration_archive_test.go b/internal/adapters/gitmodules/migration_archive_test.go new file mode 100644 index 00000000..2986ce82 --- /dev/null +++ b/internal/adapters/gitmodules/migration_archive_test.go @@ -0,0 +1,113 @@ +package gitmodules + +import ( + "os" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestFetchMigrationVerifiesV0ProtoArchiveHash(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name, config string + files, installed map[string]string + }{ + { + name: "stripped import root", + config: "generate:\n inputs: [{directory: {path: proto, root: proto}}]\n", + files: map[string]string{"proto/example.proto": "syntax = \"proto3\";", "proto/README.md": "not archived", "LICENSE": "not archived"}, + installed: map[string]string{"example.proto": "syntax = \"proto3\";"}, + }, + { + name: "default import root", + config: "generate:\n inputs: [{directory: {path: mcp, root: .}}]\n", + files: map[string]string{"mcp/options.proto": "syntax = \"proto3\";", "README.md": "not archived"}, + installed: map[string]string{"mcp/options.proto": "syntax = \"proto3\";"}, + }, + { + name: "non-proto collision is outside the released archive", + config: "generate:\n inputs: [{directory: {path: proto, root: proto}}]\n", + files: map[string]string{"proto/example.proto": "syntax = \"proto3\";", "proto/README.md": "inside root", "README.md": "outside root"}, + installed: map[string]string{"example.proto": "syntax = \"proto3\";"}, + }, + { + name: "proto-suffixed directory keeps import name", + files: map[string]string{"api.proto/message.proto": "syntax = \"proto3\";", "api.proto/README.md": "archived directory content", "README.md": "not archived"}, + installed: map[string]string{"api.proto/message.proto": "syntax = \"proto3\";"}, + }, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + if tt.config != "" { + tt.files["easyp.yaml"] = tt.config + } + repository, commit := migrationTestRepository(t, tt.files) + legacyHash := migrationTestHash(t, tt.installed) + fetched, err := (&Cache{root: t.TempDir()}).FetchMigration(t.Context(), repository, commit, legacyHash) + require.NoError(t, err) + assert.Equal(t, commit, fetched.Lock.Commit) + assert.Equal(t, commit, fetched.Lock.Version) + assert.Equal(t, migrationTestHash(t, tt.files), fetched.Lock.Hash) + assert.NotEqual(t, legacyHash, fetched.Lock.Hash) + }) + } +} + +func TestHashMigrationProtoArchiveRejectsCollisionsAndCleansUp(t *testing.T) { + // Keep this sequential because os.MkdirTemp uses the process temp environment. + for _, tt := range []struct { + name string + files map[string]string + wantError string + }{ + {name: "file collision", files: map[string]string{"a/file.proto": "first", "b/file.proto": "second"}, wantError: "legacy file collision"}, + {name: "directory and file collision", files: map[string]string{"a/nested.proto": "file", "b/nested.proto/file.proto": "directory"}, wantError: "legacy directory/file collision"}, + } { + t.Run(tt.name, func(t *testing.T) { + tt.files["buf.work.yaml"] = "version: v1\ndirectories: [a, b]\n" + repository, _ := migrationTestRepository(t, tt.files) + files, err := trackedV1Files(t.Context(), repository) + require.NoError(t, err) + archiveTemp := t.TempDir() + for _, name := range []string{"TMPDIR", "TMP", "TEMP"} { + t.Setenv(name, archiveTemp) + } + _, err = hashMigrationProtoArchive(t.Context(), repository, files) + require.ErrorContains(t, err, tt.wantError) + entries, err := os.ReadDir(archiveTemp) + require.NoError(t, err) + assert.Empty(t, entries, "failed verification must remove its temporary archive") + }) + } +} + +func TestFetchMigrationRejectsChangedV0ArchiveSources(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name, attribute, source string + }{ + {name: "export ignored proto", attribute: "proto/ignored.proto export-ignore\n", source: "syntax = \"proto3\";"}, + {name: "export substituted proto", attribute: "proto/ignored.proto export-subst\n", source: "// $Format:%H$\nsyntax = \"proto3\";"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + files := map[string]string{ + "easyp.yaml": "generate:\n inputs: [{directory: {path: proto, root: proto}}]\n", + ".gitattributes": tt.attribute, + "proto/kept.proto": "syntax = \"proto3\";", + "proto/ignored.proto": tt.source, + } + repository, commit := migrationTestRepository(t, files) + installed := map[string]string{"kept.proto": files["proto/kept.proto"]} + if tt.name == "export substituted proto" { + installed["ignored.proto"] = strings.ReplaceAll(tt.source, "$Format:%H$", commit) + } + _, err := (&Cache{root: t.TempDir()}).FetchMigration(t.Context(), repository, commit, migrationTestHash(t, installed)) + require.ErrorContains(t, err, "source selection") + require.ErrorContains(t, err, "manual migration") + }) + } +} diff --git a/internal/adapters/gitmodules/migration_files.go b/internal/adapters/gitmodules/migration_files.go new file mode 100644 index 00000000..02a698db --- /dev/null +++ b/internal/adapters/gitmodules/migration_files.go @@ -0,0 +1,56 @@ +package gitmodules + +import ( + "context" + "fmt" + "path" + "path/filepath" + "strings" + + "github.com/easyp-tech/easyp/internal/adapters/gitindex" +) + +// migrationFiles records the snapshot and the historical proof it requires. +type migrationFiles struct { + regularFiles []string + trackedFiles []string + symlinks []string + hasSymlinks bool +} + +// migrationTrackedFiles records immutable Git modes for historical proofs. +// The current logical snapshot independently validates relevant alias targets. +func migrationTrackedFiles(ctx context.Context, checkout string) (migrationFiles, error) { + entries, err := readV1GitIndex(ctx, checkout) + if err != nil { + return migrationFiles{}, fmt.Errorf("readV1GitIndex: %w", err) + } + tracked := migrationFiles{} + for _, entry := range entries { + tracked.trackedFiles = append(tracked.trackedFiles, entry.Path) + switch entry.Mode { + case gitindex.RegularFile, gitindex.ExecutableFile: + tracked.regularFiles = append(tracked.regularFiles, entry.Path) + continue + case gitindex.Symlink: + tracked.symlinks = append(tracked.symlinks, entry.Path) + // A regular-only subset cannot prove an old whole installed tree. + tracked.hasSymlinks = true + default: + // Native snapshots omit submodules, but migration cannot prove + // their legacy contracts from this repository's regular files. + return migrationFiles{}, fmt.Errorf("unsupported non-regular Git mode in %q", entry.Raw) + } + } + return tracked, nil +} + +func migrationRootThroughSymlink(name string, roots []string) (string, bool) { + for _, root := range roots { + root = path.Clean(filepath.ToSlash(root)) + if root == name || strings.HasPrefix(root, name+"/") { + return root, true + } + } + return "", false +} diff --git a/internal/adapters/gitmodules/migration_integrity.go b/internal/adapters/gitmodules/migration_integrity.go new file mode 100644 index 00000000..6e2fd286 --- /dev/null +++ b/internal/adapters/gitmodules/migration_integrity.go @@ -0,0 +1,170 @@ +package gitmodules + +import ( + "context" + "errors" + "fmt" + "io" + "os" + "path" + "path/filepath" + "slices" + "strings" + + "golang.org/x/mod/sumdb/dirhash" +) + +// A historical lock can describe a whole installed tree or released v0's proto +// archive. Each proof must match independently at the same pinned checkout. +func verifyMigrationLegacyHash(ctx context.Context, checkout v1ModuleCheckout, source, expectedHash string, tracked migrationFiles) error { + if expectedHash == "" { + return validateMigrationInitialSelection(ctx, checkout, tracked) + } + var treeHash string + var treeErr error + if !tracked.hasSymlinks { + // Raw committed bytes are only a whole-tree proof when their recorded + // digest matches; no host checkout/filter representation is assumed. + treeHash, treeErr = hashMigrationLegacyFiles(checkout.snapshot, tracked.regularFiles) + if treeErr != nil { + treeErr = fmt.Errorf("hashMigrationLegacyFiles: %w", treeErr) + } + if treeErr == nil && treeHash == expectedHash { + return validateMigrationSelection(checkout.snapshot, tracked.regularFiles, checkout.module) + } + } + + // Dropping links cannot prove a whole-tree hash. Archive verification also + // checks proto path/content equivalence when there is no historical pin. + // A pinned archive may independently succeed despite non-proto collisions + // in the alternate whole-tree representation. + files, err := snapshotV1Files(checkout.snapshot) + if err != nil { + return errors.Join(treeErr, fmt.Errorf("snapshotV1Files: %w", err)) + } + roots, err := readMigrationLegacyRoots(checkout.snapshot, files) + if err != nil { + return errors.Join(treeErr, fmt.Errorf("readMigrationLegacyRoots: %w", err)) + } + selected, err := migrationSelectedFiles(checkout.snapshot, checkout.module) + if err != nil { + return errors.Join(treeErr, fmt.Errorf("migrationSelectedFiles: %w", err)) + } + nodes, err := readMigrationProtoArchive(ctx, checkout.dir, checkout.commit, tracked.trackedFiles) + if err != nil { + return errors.Join(treeErr, fmt.Errorf("readMigrationProtoArchive: %w", err)) + } + hashes, err := migrationArchiveHashes(ctx, nodes, roots, selected) + if err != nil { + return errors.Join(treeErr, fmt.Errorf("migrationArchiveHashes: %w", err)) + } + if slices.Contains(hashes, expectedHash) { + return nil + } + + candidates := "archive " + strings.Join(hashes, " or archive ") + if !tracked.hasSymlinks { + candidates += " or whole-tree " + treeHash + } + mismatch := fmt.Errorf("legacy hash mismatch for %s at %s: got %s, want %s", source, checkout.commit, candidates, expectedHash) + return errors.Join(treeErr, mismatch) +} + +func validateMigrationInitialSelection(ctx context.Context, checkout v1ModuleCheckout, tracked migrationFiles) error { + if migrationUsesLogicalAliases(checkout, tracked) { + err := validateMigrationLogicalOwnership(ctx, checkout, tracked) + if err != nil { + return fmt.Errorf("validateMigrationLogicalOwnership: %w", err) + } + } else { + _, err := hashMigrationLegacyFiles(checkout.snapshot, tracked.regularFiles) + if err != nil { + return fmt.Errorf("hashMigrationLegacyFiles: %w", err) + } + err = validateMigrationSelection(checkout.snapshot, tracked.regularFiles, checkout.module) + if err != nil { + return fmt.Errorf("validateMigrationSelection: %w", err) + } + } + nodes, err := readMigrationProtoArchive(ctx, checkout.dir, checkout.commit, tracked.trackedFiles) + if err != nil { + return fmt.Errorf("readMigrationProtoArchive: %w", err) + } + err = validateMigrationArchiveRegularSources(nodes, checkout.snapshot, tracked.regularFiles) + if err != nil { + return fmt.Errorf("validateMigrationArchiveRegularSources: %w", err) + } + return nil +} + +// hashMigrationLegacyFiles calculates the whole-tree legacy hash without +// creating an installed tree. All tracked files and their renamed directory +// nodes participate. Released v0 installers instead used the filtered Git +// archive reproduced by hashMigrationProtoArchive. +func hashMigrationLegacyFiles(checkout string, files []string) (string, error) { + directories := make(map[string]bool) + for _, name := range files { + if !filepath.IsLocal(filepath.FromSlash(name)) || path.Clean(name) != name || strings.Contains(name, "\\") { + return "", fmt.Errorf("unsupported tracked path %q", name) + } + if _, err := regularV1File(filepath.Join(checkout, filepath.FromSlash(name))); err != nil { + return "", fmt.Errorf("regularV1File: %w", err) + } + for directory := path.Dir(name); directory != "."; directory = path.Dir(directory) { + directories[directory] = true + } + } + roots, err := readMigrationLegacyRoots(checkout, files) + if err != nil { + return "", fmt.Errorf("readMigrationLegacyRoots: %w", err) + } + + installedDirectories := make(map[string]bool) + for directory := range directories { + for destination := renameMigrationLegacyFile(directory, roots); destination != "."; destination = path.Dir(destination) { + installedDirectories[destination] = true + } + } + originals := make(map[string]string, len(files)) + names := make([]string, 0, len(files)) + for _, original := range files { + name := renameMigrationLegacyFile(original, roots) + if other, exists := originals[name]; exists { + return "", fmt.Errorf("legacy file collision at %q between %q and %q", name, other, original) + } + originals[name] = original + names = append(names, name) + for directory := path.Dir(name); directory != "."; directory = path.Dir(directory) { + installedDirectories[directory] = true + } + } + slices.Sort(names) + for _, name := range names { + if installedDirectories[name] { + return "", fmt.Errorf("legacy directory/file collision at %q from %q", name, originals[name]) + } + } + hash, err := dirhash.Hash1(names, func(name string) (io.ReadCloser, error) { + file, err := os.Open(filepath.Join(checkout, filepath.FromSlash(originals[name]))) + if err != nil { + return nil, fmt.Errorf("Open: %w", err) + } + return file, nil + }) + if err != nil { + return "", fmt.Errorf("Hash1: %w", err) + } + return hash, nil +} + +// The old renamer selected the first matching prefix, without sorting, +// cleaning, or repeatedly stripping roots from the resulting path. +func renameMigrationLegacyFile(name string, roots []string) string { + for _, root := range roots { + prefix := root + "/" + if strings.HasPrefix(name, prefix) { + return strings.TrimPrefix(name, prefix) + } + } + return name +} diff --git a/internal/adapters/gitmodules/migration_integrity_test.go b/internal/adapters/gitmodules/migration_integrity_test.go new file mode 100644 index 00000000..766d5014 --- /dev/null +++ b/internal/adapters/gitmodules/migration_integrity_test.go @@ -0,0 +1,83 @@ +package gitmodules + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestFetchMigrationMatchingWholeTreePinBypassesProtoArchive(t *testing.T) { + t.Parallel() + + files := map[string]string{ + ".gitattributes": "file.proto export-ignore\n", + "file.proto": "syntax = \"proto3\";\n", + "README": "part of the historical whole-tree representation\n", + } + repository, commit := migrationTestRepository(t, files) + expectedHash := migrationTestHash(t, files) + + fetched, err := (&Cache{root: t.TempDir()}).FetchMigration(t.Context(), repository, commit, expectedHash) + + require.NoError(t, err) + assert.Equal(t, commit, fetched.Lock.Commit) + assert.Equal(t, expectedHash, fetched.Lock.Hash) +} + +func TestFetchMigrationInitialResolutionRejectsWholeTreeCollision(t *testing.T) { + t.Parallel() + + repository, _ := migrationTestRepository(t, map[string]string{ + "easyp.yaml": "generate:\n inputs: [{directory: {path: proto, root: proto}}]\n", + "proto/file.proto": "syntax = \"proto3\";\n", + "proto/README": "inside root\n", + "README": "outside root\n", + }) + cacheDir := t.TempDir() + + fetched, err := (&Cache{root: cacheDir}).FetchMigration(t.Context(), repository, "", "") + + require.ErrorContains(t, err, "legacy file collision") + assert.Empty(t, fetched.Lock.Source) + migrationTestAssertNoCheckout(t, cacheDir) +} + +func TestFetchMigrationHashMismatchListsOnlyEligibleProofs(t *testing.T) { + t.Parallel() + + for _, tt := range []struct { + name string + auxiliaryLink bool + }{ + {name: "regular tree permits both proofs"}, + {name: "omitted links require archive proof", auxiliaryLink: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + files := map[string]string{ + "file.proto": "syntax = \"proto3\";\n", + "README": "retained regular content\n", + } + repository, commit := migrationTestRepository(t, files) + if tt.auxiliaryLink { + commit = migrationTestCommitSymlink(t, repository, ".bazelrc", "file.proto") + } + wrongHash := migrationTestHash(t, map[string]string{"different": "contents"}) + cacheDir := t.TempDir() + + fetched, err := (&Cache{root: cacheDir}).FetchMigration(t.Context(), repository, commit, wrongHash) + + require.ErrorContains(t, err, "legacy hash mismatch") + archiveHash := migrationTestHash(t, map[string]string{"file.proto": files["file.proto"]}) + assert.ErrorContains(t, err, "got archive "+archiveHash) + if tt.auxiliaryLink { + assert.NotContains(t, err.Error(), "whole-tree") + } else { + assert.ErrorContains(t, err, "or whole-tree "+migrationTestHash(t, files)) + } + assert.Empty(t, fetched.Lock.Source) + migrationTestAssertNoCheckout(t, cacheDir) + }) + } +} diff --git a/internal/adapters/gitmodules/migration_selection.go b/internal/adapters/gitmodules/migration_selection.go index 2622760a..d673c481 100644 --- a/internal/adapters/gitmodules/migration_selection.go +++ b/internal/adapters/gitmodules/migration_selection.go @@ -2,13 +2,16 @@ package gitmodules import ( "bytes" + "context" "fmt" "os" "path" "path/filepath" "slices" + "strings" "github.com/easyp-tech/easyp/internal/adapters/modfile" + moduleconfig "github.com/easyp-tech/easyp/internal/adapters/module_config" v1 "github.com/easyp-tech/easyp/internal/config/v1" "github.com/easyp-tech/easyp/internal/modules" ) @@ -70,17 +73,106 @@ func validateMigrationSelection(checkout string, files []string, module v1.Modul if err != nil { return fmt.Errorf("readMigrationLegacyRoots: %w", err) } - legacy := make(map[string]string) + legacy := make(map[string][]byte) for _, name := range files { if path.Ext(name) == ".proto" { - legacy[renameMigrationLegacyFile(name, roots)] = name + content, err := os.ReadFile(filepath.Join(checkout, filepath.FromSlash(name))) + if err != nil { + if os.IsNotExist(err) { + return fmt.Errorf("dependency %s source selection omits legacy file %q; manual migration is required: %w", module.Name, name, err) + } + return fmt.Errorf("ReadFile: %w", err) + } + importName := renameMigrationLegacyFile(name, roots) + if _, exists := legacy[importName]; exists { + return fmt.Errorf("dependency %s source selection collides at %q; manual migration is required", module.Name, importName) + } + legacy[importName] = content } } - sources, err := modules.ModuleSources(checkout, module) + selected, err := migrationSelectedFiles(checkout, module) + if err != nil { + return fmt.Errorf("migrationSelectedFiles: %w", err) + } + return validateMigrationSourceContents(module.Name, legacy, selected) +} + +func migrationUsesLogicalAliases(checkout v1ModuleCheckout, tracked migrationFiles) bool { + for _, name := range tracked.symlinks { + if moduleconfig.IsGitDependencyConfigFile(path.Base(name)) { + return true + } + if _, found := migrationRootThroughSymlink(name, checkout.module.Roots); found { + return true + } + info, err := os.Stat(filepath.Join(checkout.snapshot, filepath.FromSlash(name))) + if err != nil { + continue + } + if info.Mode().IsRegular() && path.Ext(name) == ".proto" && snapshotSelectedAlias(name, checkout.module) { + return true + } + if info.IsDir() { + for _, root := range checkout.module.Roots { + if v1FileWithin(name, root) { + return true + } + } + } + } + return false +} + +// New acquisitions may introduce usable logical roots that old installers could +// not represent. Validate their current import ownership, and require every raw +// legacy proto to remain covered by a selected logical source. Auxiliary aliases +// never exempt unrelated outside-root, hidden, or nested-module source files. +func validateMigrationLogicalOwnership(ctx context.Context, checkout v1ModuleCheckout, tracked migrationFiles) error { + _, err := migrationSelectedFiles(checkout.snapshot, checkout.module) + if err != nil { + return fmt.Errorf("migrationSelectedFiles: %w", err) + } + view, err := sourceV1SnapshotView(ctx, checkout.dir, checkout.commit) + if err != nil { + return fmt.Errorf("sourceV1SnapshotView: %w", err) + } + sources, err := modules.ModuleSources(checkout.snapshot, checkout.module) if err != nil { return fmt.Errorf("ModuleSources: %w", err) } - selected := make(map[string]string) + covered := make(map[string]bool) + for _, source := range sources { + err = modules.WalkProtoFiles(source.Path, func(file string) error { + logical, err := filepath.Rel(checkout.snapshot, file) + if err != nil { + return fmt.Errorf("Rel: %w", err) + } + resolved, err := view.Resolve(ctx, filepath.ToSlash(logical)) + if err != nil { + return fmt.Errorf("Resolve: %w", err) + } + covered[resolved.Path] = true + return nil + }) + if err != nil { + return fmt.Errorf("WalkProtoFiles: %w", err) + } + } + for _, name := range tracked.regularFiles { + if strings.HasSuffix(name, ".proto") && !covered[name] { + return fmt.Errorf("dependency %s source selection omits legacy file %q; manual migration is required", checkout.module.Name, name) + } + } + return nil +} + +func migrationSelectedFiles(checkout string, module v1.Module) (map[string][]byte, error) { + sources, err := modules.ModuleSources(checkout, module) + if err != nil { + return nil, fmt.Errorf("ModuleSources: %w", err) + } + selected := make(map[string][]byte) + owners := make(map[string]string) for _, source := range sources { err := modules.WalkProtoFiles(source.Path, func(file string) error { importName, err := filepath.Rel(source.Path, file) @@ -92,16 +184,24 @@ func validateMigrationSelection(checkout string, files []string, module v1.Modul return fmt.Errorf("Rel: %w", err) } importName, physicalName = filepath.ToSlash(importName), filepath.ToSlash(physicalName) - if previous, exists := selected[importName]; exists && previous != physicalName { + if previous, exists := owners[importName]; exists && previous != physicalName { return fmt.Errorf("dependency %s source selection collides at %q; manual migration is required", module.Name, importName) } - selected[importName] = physicalName + content, err := os.ReadFile(file) + if err != nil { + return fmt.Errorf("ReadFile: %w", err) + } + owners[importName], selected[importName] = physicalName, content return nil }) if err != nil { - return fmt.Errorf("WalkProtoFiles: %w", err) + return nil, fmt.Errorf("WalkProtoFiles: %w", err) } } + return selected, nil +} + +func validateMigrationSourceContents(module string, legacy, selected map[string][]byte) error { // Stable ordering makes the first actionable mismatch reproducible. names := make([]string, 0, len(legacy)+len(selected)) for name := range legacy { @@ -111,9 +211,11 @@ func validateMigrationSelection(checkout string, files []string, module v1.Modul names = append(names, name) } slices.Sort(names) - for _, name := range names { - if legacy[name] != selected[name] { - return fmt.Errorf("dependency %s source selection differs at import %q (legacy file %q, v1 file %q); manual migration is required", module.Name, name, legacy[name], selected[name]) + for _, name := range slices.Compact(names) { + old, oldExists := legacy[name] + current, currentExists := selected[name] + if oldExists != currentExists || !bytes.Equal(old, current) { + return fmt.Errorf("dependency %s source selection differs at import %q (legacy present %t, v1 present %t, same contents %t); manual migration is required", module, name, oldExists, currentExists, bytes.Equal(old, current)) } } return nil diff --git a/internal/adapters/gitmodules/migration_symlink_test.go b/internal/adapters/gitmodules/migration_symlink_test.go new file mode 100644 index 00000000..9ceea762 --- /dev/null +++ b/internal/adapters/gitmodules/migration_symlink_test.go @@ -0,0 +1,265 @@ +package gitmodules + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestFetchMigrationOmitsAuxiliarySymlinks(t *testing.T) { + t.Parallel() + + for _, target := range []string{"internal", "external", "dangling"} { + t.Run(target, func(t *testing.T) { + t.Parallel() + for _, mode := range []struct { + name string + native bool + pinned bool + }{ + {name: "legacy_initial"}, + {name: "legacy_proto_archive_pin", pinned: true}, + {name: "native_initial", native: true}, + {name: "native_pin", native: true, pinned: true}, + } { + t.Run(mode.name, func(t *testing.T) { + t.Parallel() + files := map[string]string{ + "proto/file.proto": "syntax = \"proto3\";\n", + "LICENSE": "retained regular auxiliary file\n", + } + if !mode.native { + files["easyp.yaml"] = "generate:\n inputs: [{directory: {path: proto, root: proto}}]\n" + } + repository, _ := migrationTestRepository(t, files) + if mode.native { + files["protobuf.mod"] = "module " + repository + "\nroots proto\n" + migrationTestWriteFiles(t, repository, map[string]string{"protobuf.mod": files["protobuf.mod"]}) + } + linkTarget := "../proto/file.proto" + if target != "internal" { + outside := t.TempDir() + linkTarget = filepath.Join(outside, "file.proto") + if target == "external" { + // The pointer is auxiliary even when its target is a proto. + // Its bytes must never enter the migration or native digest. + require.NoError(t, os.WriteFile(linkTarget, []byte("unread external proto"), 0o000)) + } + } + commit := migrationTestCommitSymlink(t, repository, "example-workspace/.bazelrc", linkTarget) + var version, legacyHash string + if mode.pinned { + version = commit + if !mode.native { + legacyHash = migrationTestHash(t, map[string]string{"file.proto": files["proto/file.proto"]}) + } + } + cacheDir := t.TempDir() + fetched, err := (&Cache{root: cacheDir}).FetchMigration(t.Context(), repository, version, legacyHash) + require.NoError(t, err) + assert.Equal(t, commit, fetched.Lock.Commit) + assert.Equal(t, commit, fetched.Lock.Version) + assert.Equal(t, []string{"proto"}, fetched.Module.Roots) + expectedSnapshot := make(map[string]string, len(files)+1) + for name, data := range files { + expectedSnapshot[name] = data + } + if target == "internal" { + expectedSnapshot["example-workspace/.bazelrc"] = files["proto/file.proto"] + } + assert.Equal(t, migrationTestHash(t, expectedSnapshot), fetched.Lock.Hash) + ordinary, err := (&Cache{root: t.TempDir()}).Fetch(t.Context(), repository, commit) + require.NoError(t, err) + assert.Equal(t, ordinary.Lock.Hash, fetched.Lock.Hash) + migrationTestAssertNoCheckout(t, cacheDir) + }) + } + }) + } +} + +func TestFetchMigrationAuxiliarySymlinksRequireProtoArchiveHash(t *testing.T) { + t.Parallel() + + for _, tt := range []struct { + name string + regularPin bool + }{ + {name: "regular_only_tree_is_not_historical_proof", regularPin: true}, + {name: "wrong_proto_archive_hash"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + files := map[string]string{"file.proto": "syntax = \"proto3\";", "README": "tracked regular auxiliary content"} + repository, _ := migrationTestRepository(t, files) + commit := migrationTestCommitSymlink(t, repository, ".bazelrc", "file.proto") + legacyHash := migrationTestHash(t, map[string]string{"file.proto": "different proto content"}) + if tt.regularPin { + legacyHash = migrationTestHash(t, files) + } + cacheDir := t.TempDir() + fetched, err := (&Cache{root: cacheDir}).FetchMigration(t.Context(), repository, commit, legacyHash) + require.ErrorContains(t, err, "legacy hash mismatch") + assert.Empty(t, fetched.Lock.Source) + migrationTestAssertNoCheckout(t, cacheDir) + }) + } +} + +func TestFetchMigrationAcceptsInitialAliasesButRequiresHistoricalPinProof(t *testing.T) { + t.Parallel() + + for _, tt := range []struct { + name string + files map[string]string + link string + target string + root string + pinErr string + }{ + {name: "proto_link", files: map[string]string{"file.proto": "proto"}, link: "alias.proto", target: "file.proto", pinErr: "legacy hash mismatch"}, + {name: "root_directory", files: map[string]string{"easyp.yaml": "generate:\n inputs: [{directory: {path: proto, root: proto}}]\n", "real/file.proto": "proto"}, link: "proto", target: "real", pinErr: "source selection"}, + {name: "root_ancestor", files: map[string]string{"easyp.yaml": "generate:\n inputs: [{directory: {path: api/proto, root: api/proto}}]\n", "real/proto/file.proto": "proto"}, link: "api", target: "real", pinErr: "source selection"}, + {name: "native_root_directory", files: map[string]string{"real/file.proto": "proto"}, link: "proto", target: "real", root: "proto", pinErr: "cannot reproduce legacy roots"}, + {name: "native_root_ancestor", files: map[string]string{"real/proto/file.proto": "proto"}, link: "api", target: "real", root: "api/proto", pinErr: "cannot reproduce legacy roots"}, + {name: "root_metadata", files: map[string]string{"config.yaml": "generate: {}\n", "file.proto": "proto"}, link: "easyp.yaml", target: "config.yaml", pinErr: "legacy hash mismatch"}, + {name: "nested_metadata", files: map[string]string{"manifest": "direct (\n)\n", "file.proto": "proto"}, link: "nested/protobuf.mod", target: "../manifest", pinErr: "legacy hash mismatch"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + for _, mode := range []struct { + name string + pinned bool + }{{name: "initial"}, {name: "pinned", pinned: true}} { + t.Run(mode.name, func(t *testing.T) { + t.Parallel() + repository, _ := migrationTestRepository(t, tt.files) + if tt.root != "" { + migrationTestWriteFiles(t, repository, map[string]string{"protobuf.mod": "module " + repository + "\nroots " + tt.root + "\n"}) + } + commit := migrationTestCommitSymlink(t, repository, tt.link, tt.target) + var version, legacyHash string + if mode.pinned { + version, legacyHash = commit, "h1:untrusted" + } + cacheDir := t.TempDir() + fetched, err := (&Cache{root: cacheDir}).FetchMigration(t.Context(), repository, version, legacyHash) + if mode.pinned { + require.ErrorContains(t, err, tt.pinErr) + assert.Empty(t, fetched.Lock.Source) + } else { + require.NoError(t, err) + ordinary, err := (&Cache{root: t.TempDir()}).Fetch(t.Context(), repository, commit) + require.NoError(t, err) + assert.Equal(t, ordinary.Lock.Hash, fetched.Lock.Hash) + assert.Equal(t, commit, fetched.Lock.Commit) + } + migrationTestAssertNoCheckout(t, cacheDir) + }) + } + }) + } +} + +func TestMigrationTrackedFilesHandlesMaterializedSymlinks(t *testing.T) { + t.Parallel() + + for _, tt := range []struct { + name string + link string + target string + root string + }{ + {name: "auxiliary", link: "example-workspace/.bazelrc", target: "../real/file.proto", root: "."}, + {name: "proto", link: "link.proto", target: "real/file.proto", root: "."}, + {name: "root", link: "proto", target: "real", root: "proto"}, + {name: "root_ancestor", link: "api", target: "real", root: "api/proto"}, + {name: "metadata", link: "buf.lock", target: "real/file.proto", root: "."}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + repository, _ := migrationTestRepository(t, map[string]string{"real/file.proto": "proto", "README": "regular auxiliary"}) + migrationTestCommitSymlink(t, repository, tt.link, tt.target) + checkout := t.TempDir() + runTestGit(t, checkout, "clone", "--quiet", "--no-checkout", "--", repository, checkout) + runTestGit(t, checkout, "-c", "core.symlinks=false", "checkout", "--quiet", "HEAD") + info, err := os.Lstat(filepath.Join(checkout, filepath.FromSlash(tt.link))) + require.NoError(t, err) + require.True(t, info.Mode().IsRegular()) + tracked, err := migrationTrackedFiles(t.Context(), checkout) + require.NoError(t, err) + assert.True(t, tracked.hasSymlinks) + assert.Equal(t, []string{tt.link}, tracked.symlinks) + assert.ElementsMatch(t, []string{"real/file.proto", "README"}, tracked.regularFiles) + }) + } +} + +func TestFetchMigrationAuxiliarySymlinksPreserveArchiveAttributes(t *testing.T) { + t.Parallel() + + for _, tt := range []struct { + name string + attribute string + source string + }{ + {name: "export_ignore", attribute: "proto/affected.proto export-ignore\n", source: "syntax = \"proto3\";"}, + {name: "export_subst", attribute: "proto/affected.proto export-subst\n", source: "// $Format:%H$\nsyntax = \"proto3\";"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + for _, mode := range []struct { + name string + pinned bool + }{{name: "initial"}, {name: "pinned", pinned: true}} { + t.Run(mode.name, func(t *testing.T) { + t.Parallel() + files := map[string]string{ + "easyp.yaml": "generate:\n inputs: [{directory: {path: proto, root: proto}}]\n", + ".gitattributes": tt.attribute, + "proto/kept.proto": "syntax = \"proto3\";", + "proto/affected.proto": tt.source, + } + repository, _ := migrationTestRepository(t, files) + commit := migrationTestCommitSymlink(t, repository, ".bazelrc", "missing") + var version, legacyHash string + if mode.pinned { + installed := map[string]string{"kept.proto": files["proto/kept.proto"]} + if tt.name == "export_subst" { + installed["affected.proto"] = strings.ReplaceAll(tt.source, "$Format:%H$", commit) + } + version, legacyHash = commit, migrationTestHash(t, installed) + } + cacheDir := t.TempDir() + _, err := (&Cache{root: cacheDir}).FetchMigration(t.Context(), repository, version, legacyHash) + require.ErrorContains(t, err, "source selection") + require.ErrorContains(t, err, "manual migration") + migrationTestAssertNoCheckout(t, cacheDir) + }) + } + }) + } +} + +func migrationTestCommitSymlink(t *testing.T, repository, name, target string) string { + t.Helper() + link := filepath.Join(repository, filepath.FromSlash(name)) + require.NoError(t, os.MkdirAll(filepath.Dir(link), 0o755)) + require.NoError(t, os.Symlink(target, link)) + runTestGit(t, repository, "add", ".") + runTestGit(t, repository, "-c", "user.name=EasyP Test", "-c", "user.email=test@example.com", "commit", "-qm", "symlink fixture") + return strings.TrimSpace(runTestGit(t, repository, "rev-parse", "HEAD")) +} + +func migrationTestAssertNoCheckout(t *testing.T, cacheDir string) { + t.Helper() + entries, err := os.ReadDir(cacheDir) + require.NoError(t, err) + for _, entry := range entries { + assert.Equal(t, "objects", entry.Name(), "temporary checkout was left behind") + } +} diff --git a/internal/adapters/gitmodules/migration_test.go b/internal/adapters/gitmodules/migration_test.go index 182d5e32..9f02e2d1 100644 --- a/internal/adapters/gitmodules/migration_test.go +++ b/internal/adapters/gitmodules/migration_test.go @@ -196,15 +196,16 @@ func TestReadMigrationLegacyRootsRejectsAmbiguity(t *testing.T) { } } -func TestFetchMigrationRejectsUnsafeNodes(t *testing.T) { +func TestFetchMigrationRejectsGitlinksAndUnprovedAliasPins(t *testing.T) { t.Parallel() tests := []struct { name string submodule bool + wantError string }{ - {name: "symlink"}, - {name: "submodule", submodule: true}, + {name: "unproved file alias pin", wantError: "legacy hash mismatch"}, + {name: "submodule", submodule: true, wantError: "unsupported non-regular Git mode"}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -213,14 +214,14 @@ func TestFetchMigrationRejectsUnsafeNodes(t *testing.T) { if tt.submodule { runTestGit(t, repository, "update-index", "--add", "--cacheinfo", "160000,"+commit+",unsafe") } else { - require.NoError(t, os.Symlink("file.proto", filepath.Join(repository, "unsafe"))) - runTestGit(t, repository, "add", "unsafe") + require.NoError(t, os.Symlink("file.proto", filepath.Join(repository, "unsafe.proto"))) + runTestGit(t, repository, "add", "unsafe.proto") } runTestGit(t, repository, "-c", "user.name=EasyP Test", "-c", "user.email=test@example.com", "commit", "-qm", "unsafe") runTestGit(t, repository, "tag", "v1.0.0") cacheDir := t.TempDir() _, err := (&Cache{root: cacheDir}).FetchMigration(t.Context(), repository, "v1.0.0", "h1:untrusted") - require.ErrorContains(t, err, "non-regular") + require.ErrorContains(t, err, tt.wantError) entries, err := os.ReadDir(cacheDir) require.NoError(t, err) assert.Empty(t, entries) @@ -228,16 +229,20 @@ func TestFetchMigrationRejectsUnsafeNodes(t *testing.T) { } } -func TestMigrationTrackedFilesRejectsMaterializedSymlink(t *testing.T) { +func TestMigrationTrackedFilesRecordsMaterializedProtoSymlinkAsPointer(t *testing.T) { t.Parallel() - repository, _ := migrationTestRepository(t, map[string]string{"link": "target.proto"}) - blob := strings.TrimSpace(runTestGit(t, repository, "rev-parse", "HEAD:link")) + repository, _ := migrationTestRepository(t, map[string]string{"link.proto": "target.proto"}) + blob := strings.TrimSpace(runTestGit(t, repository, "rev-parse", "HEAD:link.proto")) // A checkout with core.symlinks=false materializes a Git symlink as a - // regular file. The index mode must still prevent legacy verification. - runTestGit(t, repository, "update-index", "--cacheinfo", "120000,"+blob+",link") - _, err := migrationTrackedFiles(t.Context(), repository) - require.ErrorContains(t, err, "non-regular Git mode") + // regular file. The immutable mode must still exclude it from a regular + // whole-tree proof and retain it as a pointer for an independent ZIP proof. + runTestGit(t, repository, "update-index", "--cacheinfo", "120000,"+blob+",link.proto") + tracked, err := migrationTrackedFiles(t.Context(), repository) + require.NoError(t, err) + assert.Empty(t, tracked.regularFiles) + assert.Equal(t, []string{"link.proto"}, tracked.symlinks) + assert.True(t, tracked.hasSymlinks) } func TestFetchMigrationPinnedCommitIgnoresMovedTag(t *testing.T) { diff --git a/internal/adapters/gitmodules/object_cache.go b/internal/adapters/gitmodules/object_cache.go index 86b1a98c..772567ec 100644 --- a/internal/adapters/gitmodules/object_cache.go +++ b/internal/adapters/gitmodules/object_cache.go @@ -8,7 +8,6 @@ import ( "strings" "time" - moduleconfig "github.com/easyp-tech/easyp/internal/adapters/module_config" v1 "github.com/easyp-tech/easyp/internal/config/v1" ) @@ -27,7 +26,12 @@ func checkoutCachedCommit(ctx context.Context, checkout string, entry v1.LockedM } defer unlock() if _, err := os.Stat(filepath.Join(repository, "HEAD")); os.IsNotExist(err) { - if _, err := gitV1(ctx, "", "init", "--quiet", "--bare", repository); err != nil { + args := []string{"init", "--quiet", "--bare"} + if len(entry.Commit) == 64 { + args = append(args, "--object-format=sha256") + } + args = append(args, repository) + if _, err := gitV1(ctx, "", args...); err != nil { return false, err } if _, err := gitV1(ctx, repository, "config", "gc.auto", "0"); err != nil { @@ -76,10 +80,7 @@ func checkoutCachedCommit(ctx context.Context, checkout string, entry v1.LockedM if _, err := gitV1(ctx, "", "clone", "--quiet", "--shared", "--no-checkout", "--", repository, checkout); err != nil { return true, fmt.Errorf("clone cached objects: %w", err) } - if _, err := gitV1(ctx, checkout, "checkout", "--quiet", "--detach", commit); err != nil { - return true, err - } - if _, err := moduleconfig.ReadGitDependencyAt(checkout, entry.Source, candidate.subdir); err != nil { + if _, err := gitV1(ctx, checkout, "read-tree", commit); err != nil { return true, err } return true, nil diff --git a/internal/adapters/gitmodules/snapshot.go b/internal/adapters/gitmodules/snapshot.go new file mode 100644 index 00000000..c65c455f --- /dev/null +++ b/internal/adapters/gitmodules/snapshot.go @@ -0,0 +1,584 @@ +package gitmodules + +import ( + "context" + "errors" + "fmt" + "io" + "io/fs" + "os" + "path" + "path/filepath" + "strings" + + "github.com/go-git/go-billy/v5/osfs" + "github.com/go-git/go-git/v5" + "github.com/go-git/go-git/v5/plumbing" + "github.com/go-git/go-git/v5/plumbing/cache" + "github.com/go-git/go-git/v5/storage/filesystem" + + "github.com/easyp-tech/easyp/internal/adapters/gitsnapshot" + moduleconfig "github.com/easyp-tech/easyp/internal/adapters/module_config" + v1 "github.com/easyp-tech/easyp/internal/config/v1" + "github.com/easyp-tech/easyp/internal/sourceview" +) + +// prepareV1Snapshot constructs the regular logical view of one pinned Git tree. +// Metadata is materialized before passing it to the existing metadata readers. +// The caller owns and removes the returned directory. +func prepareV1Snapshot(ctx context.Context, checkout, commit, source, subdir string) (directory string, module v1.Module, err error) { + tree, err := sourceV1SnapshotTree(ctx, checkout, commit) + if err != nil { + return "", v1.Module{}, fmt.Errorf("sourceV1SnapshotTree: %w", err) + } + view := sourceview.New(tree) + directory, err = os.MkdirTemp(filepath.Dir(checkout), "snapshot-*") + if err != nil { + return "", v1.Module{}, fmt.Errorf("MkdirTemp: %w", err) + } + stagePath := directory + defer func() { + if err != nil { + _ = os.RemoveAll(stagePath) + } + }() + inventory, err := stageSnapshotMetadata(ctx, tree, view, directory, subdir) + if err != nil { + return "", v1.Module{}, fmt.Errorf("stageSnapshotMetadata: %w", err) + } + aliases, regular, metadataFailures := inventory.aliases, inventory.regular, inventory.metadataFailures + module, err = moduleconfig.ReadGitDependencyAt(directory, source, subdir) + if err != nil { + for _, failure := range metadataFailures { + if strings.Contains(err.Error(), filepath.Join(directory, filepath.FromSlash(failure.name))) { + err = errors.Join(err, fmt.Errorf("Resolve: %s: %w", failure.name, failure.err)) + } + } + return "", v1.Module{}, fmt.Errorf("ReadGitDependencyAt: %w", err) + } + ignoredMetadata := make(map[string]bool, len(metadataFailures)) + for _, failure := range metadataFailures { + if snapshotFailedMetadataOwnsSources(failure.name, directory, module) || (failure.unstaged && snapshotNativeCandidate(failure.name, source, subdir)) { + return "", v1.Module{}, fmt.Errorf("Resolve: %s: %w", failure.name, failure.err) + } + ignoredMetadata[failure.name] = true + if failure.unstaged { + continue + } + if err := os.Remove(filepath.Join(directory, filepath.FromSlash(failure.name))); err != nil { + return "", v1.Module{}, fmt.Errorf("Remove: %w", err) + } + if err := pruneEmptySnapshotParents(directory, failure.name); err != nil { + return "", v1.Module{}, fmt.Errorf("pruneEmptySnapshotParents: %w", err) + } + } + // Stage ordinary files only after metadata has selected Buf source paths. + // Discarded proto names must not collide with retained names on the host. + for _, name := range regular { + if path.Ext(name) == ".proto" && len(selectV1ProtoFiles([]string{name}, module.ProtoFilters)) == 0 { + continue + } + if err := materializeSnapshotFile(ctx, view, directory, name); err != nil { + return "", v1.Module{}, fmt.Errorf("materializeSnapshotFile: %w", err) + } + } + if err := materializeSnapshotAuxiliaryAliases(ctx, view, directory, aliases); err != nil { + return "", v1.Module{}, fmt.Errorf("materializeSnapshotAuxiliaryAliases: %w", err) + } + if err := materializeSelectedSnapshotAliases(ctx, view, directory, module, ignoredMetadata); err != nil { + return "", v1.Module{}, fmt.Errorf("materializeSelectedSnapshotAliases: %w", err) + } + // Buf filters select the same logical paths for hashing and installation. + err = filepath.WalkDir(directory, func(file string, entry fs.DirEntry, walkErr error) error { + if walkErr != nil { + return walkErr + } + if entry.IsDir() || filepath.Ext(file) != ".proto" { + return nil + } + name, err := filepath.Rel(directory, file) + if err != nil { + return fmt.Errorf("Rel: %w", err) + } + if len(selectV1ProtoFiles([]string{filepath.ToSlash(name)}, module.ProtoFilters)) == 0 { + return os.Remove(file) + } + return nil + }) + if err != nil { + return "", v1.Module{}, fmt.Errorf("WalkDir: %w", err) + } + return directory, module, nil +} + +type snapshotInventory struct { + aliases []string + regular []string + metadataFailures []snapshotMetadataFailure +} + +// pruneEmptySnapshotParents removes namespaces created only by ignored markers. +// Retained metadata stops pruning; later source files choose their own spelling. +func pruneEmptySnapshotParents(directory, name string) error { + for parent := path.Dir(name); parent != "."; parent = path.Dir(parent) { + filename := filepath.Join(directory, filepath.FromSlash(parent)) + entries, err := os.ReadDir(filename) + if err != nil { + return fmt.Errorf("ReadDir: %w", err) + } + if len(entries) > 0 { + return nil + } + if err := os.Remove(filename); err != nil { + return fmt.Errorf("Remove: %w", err) + } + } + return nil +} + +// stageSnapshotMetadata discovers immutable metadata before source filtering. +// Failed markers preserve ownership and parser precedence without pointer reads. +func stageSnapshotMetadata(ctx context.Context, tree *gitsnapshot.FS, view *sourceview.View, directory, subdir string) (snapshotInventory, error) { + var aliases, regular []string + var metadataFailures []snapshotMetadataFailure + failedMarker := func(name string, cause error) error { + failure := snapshotMetadataFailure{name: name, err: cause} + err := makeSnapshotDirectory(directory, name) + if errors.Is(err, gitsnapshot.ErrPathCollision) { + // A failed alias has no retained bytes. Keep its witness separate + // if the host cannot represent it beside staged metadata. + failure.unstaged = true + } else if err != nil { + return fmt.Errorf("makeSnapshotDirectory: %w", err) + } + metadataFailures = append(metadataFailures, failure) + return nil + } + err := fs.WalkDir(tree, ".", func(name string, entry fs.DirEntry, walkErr error) error { + if walkErr != nil { + if errors.Is(walkErr, gitsnapshot.ErrGitlink) { + return fs.SkipDir + } + return walkErr + } + if entry.Type()&fs.ModeSymlink != 0 { + aliases = append(aliases, name) + return nil + } + if entry.IsDir() { + _, statErr := tree.Lstat(name) + if statErr != nil && errors.Is(statErr, gitsnapshot.ErrGitlink) { + return fs.SkipDir + } + if statErr != nil { + return statErr + } + return nil + } + if !moduleconfig.IsGitDependencyConfigFile(path.Base(name)) { + regular = append(regular, name) + return nil + } + return materializeSnapshotFile(ctx, view, directory, name) + }) + if err != nil { + return snapshotInventory{}, fmt.Errorf("WalkDir: %w", err) + } + // Traverse directory aliases for metadata discovery. Irrelevant dangling or + // recursive aliases are omitted; roots selected by metadata are strict below. + for _, name := range aliases { + if snapshotMetadataExcluded(name, subdir) { + continue + } + resolution, resolveErr := view.Resolve(ctx, name) + if resolveErr != nil { + if moduleconfig.IsGitDependencyConfigFile(path.Base(name)) { + // A directory records presence without feeding a pointer or target bytes + // to a reader. Existing metadata precedence decides whether it is read. + if err := failedMarker(name, resolveErr); err != nil { + return snapshotInventory{}, err + } + } + continue + } + if resolution.Info.IsDir() && moduleconfig.IsGitDependencyConfigFile(path.Base(name)) { + if err := failedMarker(name, sourceview.ErrUnsupported); err != nil { + return snapshotInventory{}, err + } + continue + } + if resolution.Info.IsDir() { + walkErr := view.Walk(ctx, name, func(logical string, resolved sourceview.Resolution, walkErr error) error { + if walkErr != nil { + if moduleconfig.IsGitDependencyConfigFile(path.Base(logical)) && !snapshotMetadataExcluded(logical, subdir) { + if err := failedMarker(logical, walkErr); err != nil { + return err + } + } + return nil + } + if resolved.Info.IsDir() { + if snapshotMetadataExcluded(logical, subdir) { + return fs.SkipDir + } + return nil + } + if moduleconfig.IsGitDependencyConfigFile(path.Base(logical)) && !snapshotMetadataExcluded(logical, subdir) { + return materializeSnapshotFile(ctx, view, directory, logical) + } + return nil + }) + if walkErr != nil { + return snapshotInventory{}, fmt.Errorf("Walk: %w", walkErr) + } + continue + } + if moduleconfig.IsGitDependencyConfigFile(path.Base(name)) { + if err := materializeSnapshotFile(ctx, view, directory, name); err != nil { + return snapshotInventory{}, fmt.Errorf("materializeSnapshotFile: %w", err) + } + } + } + return snapshotInventory{aliases: aliases, regular: regular, metadataFailures: metadataFailures}, nil +} + +// materializeSelectedSnapshotAliases validates declared roots and copies aliases +// selected by source ownership and Buf filters. Auxiliary links are handled separately. +func materializeSelectedSnapshotAliases(ctx context.Context, view *sourceview.View, directory string, module v1.Module, ignoredMetadata map[string]bool) error { + for _, root := range module.Roots { + logicalRoot := filepath.ToSlash(root) + resolution, resolveErr := view.Resolve(ctx, logicalRoot) + if resolveErr != nil { + if errors.Is(resolveErr, fs.ErrNotExist) && len(resolution.Links) == 0 { + if err := makeSnapshotDirectory(directory, root); err != nil { + return fmt.Errorf("MkdirAll: %w", err) + } + continue + } + return fmt.Errorf("Resolve: root %q: %w", root, resolveErr) + } + if !resolution.Info.IsDir() { + return fmt.Errorf("module %s has invalid root %q: not a directory", module.Name, root) + } + if err := makeSnapshotDirectory(directory, root); err != nil { + return fmt.Errorf("MkdirAll: %w", err) + } + err := view.Walk(ctx, logicalRoot, func(logical string, resolved sourceview.Resolution, walkErr error) error { + if ignoredMetadata[logical] { + return nil + } + if walkErr != nil { + if logical != logicalRoot && snapshotExcludedDirectory(logical, logicalRoot, directory, module) { + return nil + } + // An unentered submodule is an opaque boundary, like a nested + // module. Explicit roots and aliases cannot cross that boundary. + if errors.Is(walkErr, gitsnapshot.ErrGitlink) && logical != logicalRoot && len(resolved.Links) == 0 { + return nil + } + if snapshotSelectedAlias(logical, module) || snapshotStrictDirectory(logical, module) || (resolved.Info != nil && resolved.Info.IsDir()) || errors.Is(walkErr, gitsnapshot.ErrGitlink) { + return walkErr + } + return nil + } + if resolved.Info.IsDir() { + if logical != logicalRoot && snapshotExcludedDirectory(logical, logicalRoot, directory, module) { + return fs.SkipDir + } + return nil + } + if len(resolved.Links) == 0 { + return nil + } + if !snapshotSelectedAlias(logical, module) { + return nil + } + return materializeSnapshotFile(ctx, view, directory, logical) + }) + if err != nil { + return fmt.Errorf("Walk: root %q: %w", root, err) + } + } + return nil +} + +func sourceV1SnapshotView(ctx context.Context, checkout, commit string) (*sourceview.View, error) { + tree, err := sourceV1SnapshotTree(ctx, checkout, commit) + if err != nil { + return nil, fmt.Errorf("sourceV1SnapshotTree: %w", err) + } + return sourceview.New(tree), nil +} + +func sourceV1SnapshotTree(ctx context.Context, checkout, commit string) (*gitsnapshot.FS, error) { + if len(commit) == 64 { + tree, err := gitsnapshot.NewRepository(ctx, checkout, commit) + if err != nil { + return nil, fmt.Errorf("NewRepository: %w", err) + } + return tree, nil + } + storage := filesystem.NewStorageWithOptions(osfs.New(filepath.Join(checkout, ".git")), cache.NewObjectLRUDefault(), filesystem.Options{AlternatesFS: osfs.New(string(filepath.Separator))}) + repo, err := git.Open(storage, osfs.New(checkout)) + if err != nil { + return nil, fmt.Errorf("Open: %w", err) + } + tree, err := gitsnapshot.New(repo, plumbing.NewHash(commit)) + if err != nil { + return nil, fmt.Errorf("New: %w", err) + } + return tree, nil +} + +type snapshotMetadataFailure struct { + name string + err error + unstaged bool +} + +// Native candidate manifests are read even when another candidate matched. +// Buf/EasyP fallback metadata has no role once a native manifest wins. +func snapshotNativeCandidate(name, source, subdir string) bool { + if path.Base(name) != v1.ModuleFile { + return false + } + directory := path.Dir(name) + if directory == path.Clean(subdir) || (subdir == "" && directory == ".") { + return true + } + major, err := v1.ModulePathMajor(source) + return err == nil && strings.HasPrefix(major, "/") && directory == path.Join(subdir, strings.TrimPrefix(major, "/")) +} + +func snapshotMetadataExcluded(name, subdir string) bool { + directory := path.Dir(name) + base := filepath.ToSlash(subdir) + if base != "" && base != "." && v1FileWithin(directory, base) { + directory = strings.TrimPrefix(strings.TrimPrefix(directory, base), "/") + } + for _, component := range strings.Split(directory, "/") { + if component != "." && (strings.HasPrefix(component, ".") || component == "easyp_vendor") { + return true + } + } + return false +} + +func snapshotFailedMetadataOwnsSources(name, directory string, module v1.Module) bool { + switch path.Base(name) { + case v1.ModuleFile, "buf.yaml", "buf.work.yaml": + default: + return false + } + parent := path.Dir(name) + for _, root := range module.Roots { + root = filepath.ToSlash(root) + if parent == root || !v1FileWithin(parent, root) { + continue + } + excluded := false + for ancestor := parent; ancestor != root && v1FileWithin(ancestor, root); ancestor = path.Dir(ancestor) { + if snapshotExcludedDirectory(ancestor, root, directory, module) { + excluded = true + break + } + } + if !excluded { + return true + } + } + return false +} + +// materializeSnapshotAuxiliaryAliases retains bounded regular non-proto aliases. +// Consumers can select arbitrary policy fragments; Fetch has no consumer context +// with which to narrow those paths. Failed auxiliary aliases remain omitted. +func materializeSnapshotAuxiliaryAliases(ctx context.Context, view *sourceview.View, directory string, aliases []string) error { + for _, name := range aliases { + resolved, err := view.Resolve(ctx, name) + if err != nil { + if ctx.Err() != nil { + return ctx.Err() + } + continue + } + if resolved.Info.IsDir() { + err = view.Walk(ctx, name, func(logical string, resolved sourceview.Resolution, walkErr error) error { + if walkErr != nil { + return nil + } + if resolved.Info.IsDir() || path.Ext(logical) == ".proto" { + return nil + } + return materializeSnapshotFile(ctx, view, directory, logical) + }) + if err != nil { + return fmt.Errorf("Walk: %w", err) + } + continue + } + if path.Ext(name) == ".proto" { + continue + } + if err := materializeSnapshotFile(ctx, view, directory, name); err != nil { + return fmt.Errorf("materializeSnapshotFile: %w", err) + } + } + return nil +} + +func snapshotSelectedAlias(name string, module v1.Module) bool { + if moduleconfig.IsGitDependencyConfigFile(path.Base(name)) { + return true + } + if path.Ext(name) != ".proto" { + return false + } + for _, root := range module.Roots { + if v1FileWithin(name, root) { + return len(selectV1ProtoFiles([]string{name}, module.ProtoFilters)) != 0 + } + } + return false +} + +func snapshotStrictDirectory(name string, module v1.Module) bool { + for _, root := range module.Roots { + if name == filepath.ToSlash(root) { + return true + } + } + for _, filter := range module.ProtoFilters { + for _, included := range filter.Includes { + if name == filepath.ToSlash(included) { + return true + } + } + } + return false +} + +func snapshotExcludedDirectory(name, root, directory string, module v1.Module) bool { + relative := strings.TrimPrefix(strings.TrimPrefix(name, root), "/") + for _, component := range strings.Split(relative, "/") { + if strings.HasPrefix(component, ".") || component == "easyp_vendor" { + return true + } + } + // Nested native modules retain their own ownership. + if name != root { + data, err := os.ReadFile(filepath.Join(directory, filepath.FromSlash(name), v1.ModuleFile)) + if err == nil && v1.IsModuleManifest(data) { + return true + } + } + if len(module.ProtoFilters) == 0 { + return false + } + for _, filter := range module.ProtoFilters { + if !v1FileWithin(name, filter.Root) { + continue + } + allowed := len(filter.Includes) == 0 + for _, included := range filter.Includes { + allowed = allowed || v1FileWithin(name, included) || v1FileWithin(included, name) + } + for _, excluded := range filter.Excludes { + if v1FileWithin(name, excluded) { + allowed = false + break + } + } + if allowed { + return false + } + } + return true +} + +func materializeSnapshotFile(ctx context.Context, view *sourceview.View, directory, name string) error { + file, err := view.Open(ctx, name) + if err != nil { + return fmt.Errorf("Open: %w", err) + } + info, err := file.Stat() + if err != nil { + _ = file.Close() + return fmt.Errorf("Stat: %w", err) + } + if !info.Mode().IsRegular() { + _ = file.Close() + return fmt.Errorf("non-regular snapshot file %q", name) + } + destination := filepath.Join(directory, filepath.FromSlash(name)) + if err := gitsnapshot.ValidateDestination(directory, name); err != nil { + closeErr := file.Close() + return errors.Join(fmt.Errorf("ValidateDestination: %w", err), closeErr) + } + if err := os.MkdirAll(filepath.Dir(destination), 0o755); err != nil { + _ = file.Close() + return fmt.Errorf("MkdirAll: %w", err) + } + output, err := os.OpenFile(destination, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, info.Mode().Perm()) + if err != nil { + _ = file.Close() + return fmt.Errorf("OpenFile: %w", err) + } + _, copyErr := io.Copy(output, file) + closeErr := errors.Join(file.Close(), output.Close()) + if err := errors.Join(copyErr, closeErr, ctx.Err()); err != nil { + return fmt.Errorf("Copy: %w", err) + } + return nil +} + +func makeSnapshotDirectory(directory, name string) error { + if err := gitsnapshot.ValidateDestination(directory, name); err != nil { + return fmt.Errorf("ValidateDestination: %w", err) + } + if err := os.MkdirAll(filepath.Join(directory, filepath.FromSlash(name)), 0o755); err != nil { + return fmt.Errorf("MkdirAll: %w", err) + } + return nil +} + +func snapshotV1Files(directory string) ([]string, error) { + var files []string + err := filepath.WalkDir(directory, func(file string, entry fs.DirEntry, walkErr error) error { + if walkErr != nil { + return walkErr + } + if entry.IsDir() { + return nil + } + info, err := entry.Info() + if err != nil { + return fmt.Errorf("Info: %w", err) + } + if !info.Mode().IsRegular() { + return fmt.Errorf("non-regular snapshot file %q", file) + } + name, err := filepath.Rel(directory, file) + if err != nil { + return fmt.Errorf("Rel: %w", err) + } + files = append(files, filepath.ToSlash(name)) + return nil + }) + if err != nil { + return nil, fmt.Errorf("WalkDir: %w", err) + } + return files, nil +} + +func hashSnapshotV1Files(directory string) (string, error) { + files, err := snapshotV1Files(directory) + if err != nil { + return "", fmt.Errorf("snapshotV1Files: %w", err) + } + hash, err := hashV1Files(directory, files) + if err != nil { + return "", fmt.Errorf("hashV1Files: %w", err) + } + return hash, nil +} diff --git a/internal/adapters/gitmodules/snapshot_checkout.go b/internal/adapters/gitmodules/snapshot_checkout.go new file mode 100644 index 00000000..ac35d4a1 --- /dev/null +++ b/internal/adapters/gitmodules/snapshot_checkout.go @@ -0,0 +1,148 @@ +package gitmodules + +import ( + "context" + "fmt" + "os" + "strings" + + moduleconfig "github.com/easyp-tech/easyp/internal/adapters/module_config" + v1 "github.com/easyp-tech/easyp/internal/config/v1" + "github.com/easyp-tech/easyp/internal/modules" +) + +// Fetch resolves a revision to the current immutable logical snapshot policy. +func (c *Cache) Fetch(ctx context.Context, source, version string) (modules.Fetched, error) { + checkout, err := checkoutV1Module(ctx, source, version, c.root) + if err != nil { + return modules.Fetched{}, fmt.Errorf("checkoutV1Module: %w", err) + } + defer func() { _ = os.RemoveAll(checkout.dir); _ = os.RemoveAll(checkout.snapshot) }() + if err := moduleconfig.ValidateLegacyMajor(checkout.snapshot, source, version); err != nil { + return modules.Fetched{}, fmt.Errorf("ValidateLegacyMajor: %w", err) + } + hash, err := hashSnapshotV1Files(checkout.snapshot) + if err != nil { + return modules.Fetched{}, fmt.Errorf("hashSnapshotV1Files: %w", err) + } + if version == "" { + version = checkout.commit + } + module, bindings, err := c.resolveBSRDependencies(ctx, checkout.module) + if err != nil { + return modules.Fetched{}, fmt.Errorf("resolveBSRDependencies: %w", err) + } + return modules.Fetched{Module: module, Lock: v1.LockedModule{Source: source, Version: version, Commit: checkout.commit, Hash: hash, BSR: bindings}}, nil +} + +// checkoutV1Module retains Git objects and an index for historical proofs and a +// bounded regular logical snapshot for all current metadata and hashing. +func checkoutV1Module(ctx context.Context, source, version, cacheRoot string) (checkout v1ModuleCheckout, err error) { + if err := v1.ValidateModuleVersion(source, version); err != nil { + return v1ModuleCheckout{}, fmt.Errorf("ValidateModuleVersion: %w", err) + } + if err := os.MkdirAll(cacheRoot, 0o755); err != nil { + return v1ModuleCheckout{}, fmt.Errorf("MkdirAll: %w", err) + } + var candidates []v1GitModuleCandidate + if version != "" && !v1.IsCommitRef(version) { + candidate, err := findV1GitModuleTag(ctx, source, version) + if err != nil { + return v1ModuleCheckout{}, fmt.Errorf("findV1GitModuleTag: %w", err) + } + candidates = []v1GitModuleCandidate{candidate} + } else { + candidates, err = v1GitModuleCandidates(source) + if err != nil { + return v1ModuleCheckout{}, fmt.Errorf("v1GitModuleCandidates: %w", err) + } + } + var firstErr, moduleErr error + for _, candidate := range candidates { + if err := ctx.Err(); err != nil { + return v1ModuleCheckout{}, fmt.Errorf("Err: %w", err) + } + dir, err := os.MkdirTemp(cacheRoot, "git-*") + if err != nil { + return v1ModuleCheckout{}, fmt.Errorf("MkdirTemp: %w", err) + } + checkout, cloned, candidateErr := checkoutSnapshotCandidate(ctx, dir, source, version, candidate) + if candidateErr == nil { + return checkout, nil + } + _ = os.RemoveAll(dir) + if firstErr == nil { + firstErr = candidateErr + } + if cloned { + moduleErr = candidateErr + } + } + if moduleErr != nil { + return v1ModuleCheckout{}, fmt.Errorf("%s: %w", source, moduleErr) + } + if v1.IsCommitRef(version) { + return v1ModuleCheckout{}, fmt.Errorf("%s: could not fetch locked commit %s; check repository access before changing the lock: %w", source, version, firstErr) + } + return v1ModuleCheckout{}, fmt.Errorf("could not fetch Git module %s@%s: %w", source, version, firstErr) +} + +func checkoutSnapshotCandidate(ctx context.Context, dir, source, version string, candidate v1GitModuleCandidate) (v1ModuleCheckout, bool, error) { + if v1.IsCommitRef(version) { + cloned, err := checkoutCachedCommit(ctx, dir, v1.LockedModule{Source: source, Commit: version}, candidate) + if err != nil { + return v1ModuleCheckout{}, cloned, err + } + } else { + args := []string{"clone", "--quiet", "--depth=1", "--no-checkout"} + if version != "" { + args = append(args, "--branch", candidate.tag(strings.TrimSuffix(version, "+incompatible"))) + } + args = append(args, "--", candidate.remote, dir) + if _, err := gitV1(ctx, "", args...); err != nil { + return v1ModuleCheckout{}, false, fmt.Errorf("gitV1: %w", err) + } + } + revision := "HEAD" + if version != "" && !v1.IsCommitRef(version) { + revision = "refs/tags/" + candidate.tag(strings.TrimSuffix(version, "+incompatible")) + "^{commit}" + } + commit, err := gitV1(ctx, dir, "rev-parse", "--verify", revision) + if err != nil { + return v1ModuleCheckout{}, true, fmt.Errorf("gitV1: %w", err) + } + commit = strings.TrimSpace(commit) + if _, err := gitV1(ctx, dir, "read-tree", commit); err != nil { + return v1ModuleCheckout{}, true, fmt.Errorf("gitV1: %w", err) + } + stage, module, err := prepareV1Snapshot(ctx, dir, commit, source, candidate.subdir) + if err != nil { + return v1ModuleCheckout{}, true, fmt.Errorf("prepareV1Snapshot: %w", err) + } + return v1ModuleCheckout{dir: dir, module: module, commit: commit, snapshot: stage}, true, nil +} + +func fetchPinnedV1Module(ctx context.Context, entry v1.LockedModule, cacheRoot, installed string) error { + checkout, err := checkoutV1Module(ctx, entry.Source, entry.Commit, cacheRoot) + if err != nil { + return fmt.Errorf("checkoutV1Module: %w", err) + } + defer func() { _ = os.RemoveAll(checkout.dir); _ = os.RemoveAll(checkout.snapshot) }() + actual, err := hashSnapshotV1Files(checkout.snapshot) + if err != nil { + return fmt.Errorf("hashSnapshotV1Files: %w", err) + } + if !strings.EqualFold(checkout.commit, entry.Commit) { + return fmt.Errorf("%s: snapshot commit does not match lock", entry.Source) + } + if actual != entry.Hash { + return fmt.Errorf("%s@%s hash mismatch: got %s, want %s; downloaded contents do not match the pinned hash; investigate repository integrity and keep protobuf.lock unchanged", entry.Source, entry.Commit, actual, entry.Hash) + } + if err := moduleconfig.ValidateLegacyMajor(checkout.snapshot, entry.Source, entry.Version); err != nil { + return fmt.Errorf("ValidateLegacyMajor: %w", err) + } + if err := os.Rename(checkout.snapshot, installed); err != nil { + return fmt.Errorf("Rename: %w", err) + } + return nil +} diff --git a/internal/adapters/gitmodules/snapshot_collision_test.go b/internal/adapters/gitmodules/snapshot_collision_test.go new file mode 100644 index 00000000..c89400fb --- /dev/null +++ b/internal/adapters/gitmodules/snapshot_collision_test.go @@ -0,0 +1,178 @@ +package gitmodules + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + v1 "github.com/easyp-tech/easyp/internal/config/v1" +) + +func TestSnapshotRejectsHostFilenameCollisions(t *testing.T) { + t.Parallel() + repository := t.TempDir() + upper := filepath.Join(repository, "Case.proto") + require.NoError(t, os.WriteFile(upper, []byte("UPPER"), 0o644)) + _, caseErr := os.Stat(filepath.Join(repository, "case.proto")) + caseInsensitive := caseErr == nil + runTestGit(t, repository, "init", "-q") + first := strings.TrimSpace(runTestGit(t, repository, "hash-object", "-w", "--", "Case.proto")) + other := filepath.Join(repository, "other") + require.NoError(t, os.WriteFile(other, []byte("LOWER"), 0o644)) + second := strings.TrimSpace(runTestGit(t, repository, "hash-object", "-w", "--", "other")) + runTestGit(t, repository, "update-index", "--add", "--cacheinfo", "100644,"+first+",Case.proto") + runTestGit(t, repository, "update-index", "--add", "--cacheinfo", "100644,"+second+",case.proto") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "case-sensitive Git names") + + fetched, err := (&Cache{root: t.TempDir()}).Fetch(t.Context(), repository, "") + + if caseInsensitive { + require.ErrorContains(t, err, "snapshot path collision") + assert.Empty(t, fetched.Lock.Source) + } else { + require.NoError(t, err) + assert.NotEmpty(t, fetched.Lock.Hash) + } +} + +func TestSnapshotCollisionsRespectRetainedPaths(t *testing.T) { + t.Parallel() + for _, tt := range []struct{ name, upper, lower, excluded string }{ + {name: "retained directory collision", upper: "Case/a.proto", lower: "case/b.proto"}, + {name: "excluded file", upper: "Case.proto", lower: "case.proto", excluded: "Case.proto"}, + {name: "excluded namespace", upper: "Case/a.proto", lower: "case/b.proto", excluded: "Case"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + repository := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(repository, "probe"), []byte("probe"), 0o644)) + _, caseErr := os.Stat(filepath.Join(repository, "Probe")) + caseInsensitive := caseErr == nil + runTestGit(t, repository, "init", "-q") + for _, file := range []struct{ name, data string }{{name: tt.upper, data: "UPPER"}, {name: tt.lower, data: "LOWER"}} { + require.NoError(t, os.WriteFile(filepath.Join(repository, "blob"), []byte(file.data), 0o644)) + hash := strings.TrimSpace(runTestGit(t, repository, "hash-object", "-w", "--", "blob")) + runTestGit(t, repository, "update-index", "--add", "--cacheinfo", "100644,"+hash+","+file.name) + } + if tt.excluded != "" { + require.NoError(t, os.WriteFile(filepath.Join(repository, "buf.yaml"), []byte("version: v1\nbuild:\n excludes: ["+tt.excluded+"]\n"), 0o644)) + runTestGit(t, repository, "add", "buf.yaml") + } + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "retained logical paths") + cache := &Cache{root: t.TempDir()} + fetched, err := cache.Fetch(t.Context(), repository, "") + if caseInsensitive && tt.excluded == "" { + require.ErrorContains(t, err, "snapshot path collision") + return + } + require.NoError(t, err) + require.NoError(t, cache.Install(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{fetched.Lock}})) + installed, _, err := cache.Cached(fetched.Lock) + require.NoError(t, err) + data, err := os.ReadFile(filepath.Join(installed, tt.lower)) + require.NoError(t, err) + assert.Equal(t, "LOWER", string(data)) + if tt.excluded != "" { + names, err := snapshotV1Files(installed) + require.NoError(t, err) + assert.NotContains(t, names, tt.upper) + assert.Contains(t, names, tt.lower) + } + }) + } +} + +func TestSnapshotIgnoredMetadataDoesNotRetainCollisionNamespace(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name string + retainedMetadata bool + }{ + {name: "without retained metadata"}, + {name: "with retained metadata", retainedMetadata: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + repository := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(repository, "protobuf.mod"), []byte("module "+repository+"\nroots case\n"), 0o644)) + require.NoError(t, os.MkdirAll(filepath.Join(repository, "case"), 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(repository, "case/file.proto"), []byte("LOWER"), 0o644)) + require.NoError(t, os.WriteFile(filepath.Join(repository, "pointer"), []byte("missing"), 0o644)) + runTestGit(t, repository, "init", "-q") + runTestGit(t, repository, "add", "protobuf.mod", "case/file.proto") + if tt.retainedMetadata { + require.NoError(t, os.WriteFile(filepath.Join(repository, "case/buf.yaml"), []byte("version: v1\n"), 0o644)) + runTestGit(t, repository, "add", "case/buf.yaml") + } + hash := strings.TrimSpace(runTestGit(t, repository, "hash-object", "-w", "--", "pointer")) + runTestGit(t, repository, "update-index", "--add", "--cacheinfo", "120000,"+hash+",Case/buf.yaml") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "ignored failure namespace") + cache := &Cache{root: t.TempDir()} + fetched, err := cache.Fetch(t.Context(), repository, "") + require.NoError(t, err) + require.NoError(t, cache.Install(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{fetched.Lock}})) + installed, _, err := cache.Cached(fetched.Lock) + require.NoError(t, err) + data, err := os.ReadFile(filepath.Join(installed, "case/file.proto")) + require.NoError(t, err) + assert.Equal(t, "LOWER", string(data)) + names, err := snapshotV1Files(installed) + require.NoError(t, err) + assert.NotContains(t, names, "Case/buf.yaml") + if tt.retainedMetadata { + assert.Contains(t, names, "case/buf.yaml") + } + }) + } +} + +func TestSnapshotUnusedAlternativeMajorMetadataDoesNotOverrideNative(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name, alias string + required bool + }{ + {name: "unused Buf", alias: "buf.yaml"}, + {name: "unused EasyP", alias: "easyp.yaml"}, + {name: "required native candidate", alias: "protobuf.mod", required: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + repository := t.TempDir() + source := repository + "/v2" + require.NoError(t, os.WriteFile(filepath.Join(repository, "protobuf.mod"), []byte("module "+source+"\nroots proto\n"), 0o644)) + for name, content := range map[string]string{"proto/file.proto": "SOURCE", "V2/buf.yaml": "version: v1\n", "pointer": "missing"} { + filename := filepath.Join(repository, name) + require.NoError(t, os.MkdirAll(filepath.Dir(filename), 0o755)) + require.NoError(t, os.WriteFile(filename, []byte(content), 0o644)) + } + runTestGit(t, repository, "init", "-q") + runTestGit(t, repository, "add", "protobuf.mod", "proto/file.proto", "V2/buf.yaml") + hash := strings.TrimSpace(runTestGit(t, repository, "hash-object", "-w", "--", "pointer")) + runTestGit(t, repository, "update-index", "--add", "--cacheinfo", "120000,"+hash+",v2/"+tt.alias) + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "native precedence over unused major metadata") + cache := &Cache{root: t.TempDir()} + fetched, err := cache.Fetch(t.Context(), source, "") + if tt.required { + require.ErrorIs(t, err, os.ErrNotExist) + assert.Empty(t, fetched.Lock.Source) + return + } + require.NoError(t, err) + assert.Equal(t, source, fetched.Module.Name) + assert.Equal(t, []string{"proto"}, fetched.Module.Roots) + require.NoError(t, cache.Install(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{fetched.Lock}})) + installed, module, err := cache.Cached(fetched.Lock) + require.NoError(t, err) + assert.Equal(t, source, module.Name) + names, err := snapshotV1Files(installed) + require.NoError(t, err) + assert.Contains(t, names, "V2/buf.yaml") + assert.NotContains(t, names, "v2/"+tt.alias) + }) + } +} diff --git a/internal/adapters/gitmodules/snapshot_sha256_test.go b/internal/adapters/gitmodules/snapshot_sha256_test.go new file mode 100644 index 00000000..2de697eb --- /dev/null +++ b/internal/adapters/gitmodules/snapshot_sha256_test.go @@ -0,0 +1,45 @@ +package gitmodules + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + v1 "github.com/easyp-tech/easyp/internal/config/v1" +) + +func TestSnapshotSHA256RepositoryAliases(t *testing.T) { + t.Parallel() + repository := t.TempDir() + require.NoError(t, os.MkdirAll(filepath.Join(repository, "real"), 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(repository, "real/file.proto"), []byte("syntax = \"proto3\"; package sha.v1;\n"), 0o644)) + require.NoError(t, os.WriteFile(filepath.Join(repository, "protobuf.mod"), []byte("module "+repository+"\nroots api\n"), 0o644)) + require.NoError(t, os.Symlink("real", filepath.Join(repository, "api"))) + runTestGit(t, repository, "init", "--quiet", "--object-format=sha256") + runTestGit(t, repository, "add", ".") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "SHA256 alias") + commit := runTestGit(t, repository, "rev-parse", "HEAD") + require.Len(t, commit[:len(commit)-1], 64) + runTestGit(t, repository, "tag", "v1.0.0") + for _, ref := range []string{"", "v1.0.0"} { + t.Run("ref="+ref, func(t *testing.T) { + t.Parallel() + cache := &Cache{root: t.TempDir()} + fetched, err := cache.Fetch(t.Context(), repository, ref) + require.NoError(t, err) + assert.Len(t, fetched.Lock.Commit, 64) + assert.Contains(t, fetched.Lock.Hash, "h1:") + lock := v1.Lock{Version: 1, Modules: []v1.LockedModule{fetched.Lock}} + require.NoError(t, cache.Install(t.Context(), lock)) + directory, _, err := cache.Cached(fetched.Lock) + require.NoError(t, err) + content, err := os.ReadFile(filepath.Join(directory, "api/file.proto")) + require.NoError(t, err) + assert.Equal(t, "syntax = \"proto3\"; package sha.v1;\n", string(content)) + require.NoError(t, cache.VerifyCached(t.Context(), lock)) + }) + } +} diff --git a/internal/adapters/gitmodules/snapshot_test.go b/internal/adapters/gitmodules/snapshot_test.go new file mode 100644 index 00000000..0eb5c86a --- /dev/null +++ b/internal/adapters/gitmodules/snapshot_test.go @@ -0,0 +1,400 @@ +package gitmodules + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/easyp-tech/easyp/internal/adapters/gitsnapshot" + v1 "github.com/easyp-tech/easyp/internal/config/v1" + "github.com/easyp-tech/easyp/internal/modules" + policyresolver "github.com/easyp-tech/easyp/internal/policy" +) + +func TestFreshSnapshotMaterializesLogicalAliases(t *testing.T) { + t.Parallel() + repository := t.TempDir() + files := map[string]string{ + "meta/module": "module " + repository + "\nroots proto\n", + "sources/file.proto": "syntax = \"proto3\";\npackage alias;\n", + } + for name, data := range files { + file := filepath.Join(repository, name) + require.NoError(t, os.MkdirAll(filepath.Dir(file), 0o755)) + require.NoError(t, os.WriteFile(file, []byte(data), 0o644)) + } + require.NoError(t, os.Symlink("meta/module", filepath.Join(repository, "protobuf.mod"))) + require.NoError(t, os.Symlink("sources", filepath.Join(repository, "proto"))) + require.NoError(t, os.Symlink("missing", filepath.Join(repository, "auxiliary"))) + runTestGit(t, repository, "init", "-q") + runTestGit(t, repository, "add", ".") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "initial") + commit := strings.TrimSpace(runTestGit(t, repository, "rev-parse", "HEAD")) + cache := &Cache{root: t.TempDir()} + source := repository + fetched, err := cache.Fetch(t.Context(), source, commit) + require.NoError(t, err) + assert.True(t, strings.HasPrefix(fetched.Lock.Hash, "h1:"), fetched.Lock.Hash) + assert.Equal(t, []string{"proto"}, fetched.Module.Roots) + require.NoError(t, cache.Install(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{fetched.Lock}})) + installed, _, err := cache.Cached(fetched.Lock) + require.NoError(t, err) + data, err := os.ReadFile(filepath.Join(installed, "proto/file.proto")) + require.NoError(t, err) + assert.Equal(t, files["sources/file.proto"], string(data)) + info, err := os.Lstat(filepath.Join(installed, "protobuf.mod")) + require.NoError(t, err) + assert.True(t, info.Mode().IsRegular()) + assert.Equal(t, v1CacheSourceKey(commit+":"+fetched.Lock.Hash), filepath.Base(installed)) +} + +func TestFreshRegularSnapshotUsesH1(t *testing.T) { + t.Parallel() + repository := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(repository, "file.proto"), []byte("syntax = \"proto3\";\n"), 0o644)) + runTestGit(t, repository, "init", "-q") + runTestGit(t, repository, "add", ".") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "initial") + fetched, err := (&Cache{root: t.TempDir()}).Fetch(t.Context(), repository, "") + require.NoError(t, err) + assert.True(t, strings.HasPrefix(fetched.Lock.Hash, "h1:"), fetched.Lock.Hash) +} + +func TestSnapshotIgnoresCheckoutTransformations(t *testing.T) { + repository := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(repository, "file.proto"), []byte("syntax = \"proto3\";\n"), 0o644)) + runTestGit(t, repository, "init", "-q") + runTestGit(t, repository, "add", ".") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "initial") + commit := strings.TrimSpace(runTestGit(t, repository, "rev-parse", "HEAD")) + global := filepath.Join(t.TempDir(), "gitconfig") + require.NoError(t, os.WriteFile(global, []byte("[core]\n autocrlf = true\n"), 0o600)) + t.Setenv("GIT_CONFIG_GLOBAL", global) + cache := &Cache{root: t.TempDir()} + fresh, err := cache.Fetch(t.Context(), repository, commit) + require.NoError(t, err) + assert.Equal(t, migrationTestHash(t, map[string]string{"file.proto": "syntax = \"proto3\";\n"}), fresh.Lock.Hash) + require.NoError(t, cache.Install(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{fresh.Lock}})) +} + +func TestSnapshotSelectedAliasesAreBounded(t *testing.T) { + t.Parallel() + tests := []struct{ name, link, target, metadata, wantErr string }{ + {name: "internal file", link: "proto/alias.proto", target: "../data/file.proto"}, + {name: "component order", link: "proto/alias.proto", target: "../jump/../file.proto"}, + {name: "selected missing", link: "proto/alias.proto", target: "missing", wantErr: "file does not exist"}, + {name: "selected external", link: "proto/alias.proto", target: "../../external", wantErr: "outside root"}, + {name: "selected absolute", link: "proto/alias.proto", target: "/tmp/external", wantErr: "outside root"}, + {name: "selected file cycle", link: "proto/alias.proto", target: "alias.proto", wantErr: "cycle"}, + {name: "root missing", link: "alias", target: "missing", metadata: "roots alias\n", wantErr: "file does not exist"}, + {name: "root external", link: "alias", target: "../external", metadata: "roots alias\n", wantErr: "outside root"}, + {name: "root cycle", link: "alias", target: ".", metadata: "roots alias\n", wantErr: "cycle"}, + {name: "outside roots omitted", link: "alias.proto", target: "missing"}, + {name: "hidden metadata omitted", link: ".hidden/protobuf.mod", target: "missing"}, + {name: "unused Buf metadata omitted", link: "buf.yaml", target: "missing"}, + {name: "unused Buf metadata beneath selected root omitted", link: "buf.yaml", target: "missing", metadata: "roots .\n"}, + {name: "excluded omitted", link: "proto/private/alias.proto", target: "missing", metadata: "buf"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + repository := t.TempDir() + manifest := "module " + repository + "\nroots proto\n" + if tt.metadata != "" && tt.metadata != "buf" { + manifest = "module " + repository + "\n" + tt.metadata + } + data := map[string]string{"protobuf.mod": manifest, "data/file.proto": "target", "data/deeper/marker": "marker", "file.proto": "wrong", "proto/regular.proto": "regular"} + if tt.metadata == "buf" { + delete(data, "protobuf.mod") + data["buf.yaml"] = "version: v1\nbuild:\n roots: [proto]\n excludes: [proto/private]\n" + } + for name, contents := range data { + file := filepath.Join(repository, name) + require.NoError(t, os.MkdirAll(filepath.Dir(file), 0o755)) + require.NoError(t, os.WriteFile(file, []byte(contents), 0o644)) + } + require.NoError(t, os.Symlink("data/deeper", filepath.Join(repository, "jump"))) + link := filepath.Join(repository, tt.link) + require.NoError(t, os.MkdirAll(filepath.Dir(link), 0o755)) + require.NoError(t, os.Symlink(tt.target, link)) + runTestGit(t, repository, "init", "-q") + runTestGit(t, repository, "add", ".") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "initial") + fetched, err := (&Cache{root: t.TempDir()}).Fetch(t.Context(), repository, "") + if tt.wantErr != "" { + require.ErrorContains(t, err, tt.wantErr) + return + } + require.NoError(t, err) + assert.True(t, strings.HasPrefix(fetched.Lock.Hash, "h1:")) + }) + } +} + +func TestSnapshotGitlinkBoundaries(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name, root, alias string + fail bool + }{ + {name: "opaque descendant", root: "proto"}, + {name: "declared root", root: "proto/submodule", fail: true}, + {name: "selected alias", root: "proto", alias: "submodule/file.proto", fail: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + repository := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(repository, "protobuf.mod"), []byte("module "+repository+"\nroots "+tt.root+"\n"), 0o644)) + require.NoError(t, os.MkdirAll(filepath.Join(repository, "proto"), 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(repository, "proto/file.proto"), []byte("file"), 0o644)) + if tt.alias != "" { + require.NoError(t, os.Symlink(tt.alias, filepath.Join(repository, "proto/alias.proto"))) + } + runTestGit(t, repository, "init", "-q") + runTestGit(t, repository, "add", ".") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "initial") + commit := strings.TrimSpace(runTestGit(t, repository, "rev-parse", "HEAD")) + runTestGit(t, repository, "update-index", "--add", "--cacheinfo", "160000", commit, "proto/submodule") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "gitlink") + fetched, err := (&Cache{root: t.TempDir()}).Fetch(t.Context(), repository, "") + if tt.fail { + require.ErrorIs(t, err, gitsnapshot.ErrGitlink) + return + } + require.NoError(t, err) + assert.NotEmpty(t, fetched.Lock.Hash) + }) + } +} + +func TestSnapshotBufWorkspaceUsesLogicalDirectoryAlias(t *testing.T) { + t.Parallel() + repository := t.TempDir() + files := map[string]string{ + "buf.work.yaml": "version: v1\ndirectories:\n - alias\n", + "physical/buf.yaml": "version: v1\n", + "physical/file.proto": "syntax = \"proto3\";\n", + } + for name, data := range files { + file := filepath.Join(repository, name) + require.NoError(t, os.MkdirAll(filepath.Dir(file), 0o755)) + require.NoError(t, os.WriteFile(file, []byte(data), 0o644)) + } + require.NoError(t, os.Symlink("physical", filepath.Join(repository, "alias"))) + require.NoError(t, os.Symlink("missing", filepath.Join(repository, "physical/aaa_auxiliary"))) + runTestGit(t, repository, "init", "-q") + runTestGit(t, repository, "add", ".") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "initial") + cache := &Cache{root: t.TempDir()} + fetched, err := cache.Fetch(t.Context(), repository, "") + require.NoError(t, err) + assert.Equal(t, []string{"alias"}, fetched.Module.Roots) + require.NoError(t, cache.Install(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{fetched.Lock}})) + installed, _, err := cache.Cached(fetched.Lock) + require.NoError(t, err) + bytes, err := os.ReadFile(filepath.Join(installed, "alias/file.proto")) + require.NoError(t, err) + assert.Equal(t, files["physical/file.proto"], string(bytes)) +} + +func TestSnapshotInstallLeavesObsoleteDirectoriesUntouched(t *testing.T) { + t.Parallel() + repository := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(repository, "file.proto"), []byte("contents"), 0o644)) + runTestGit(t, repository, "init", "-q") + runTestGit(t, repository, "add", ".") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "initial") + cache := &Cache{root: t.TempDir()} + fetched, err := cache.Fetch(t.Context(), repository, "") + require.NoError(t, err) + obsolete := filepath.Join(cache.root, "modules", v1CacheSourceKey(repository), fetched.Lock.Commit) + require.NoError(t, os.MkdirAll(obsolete, 0o755)) + sentinel := filepath.Join(obsolete, "foreign") + require.NoError(t, os.WriteFile(sentinel, []byte("keep"), 0o600)) + require.NoError(t, cache.Install(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{fetched.Lock}})) + installed, _, err := cache.Cached(fetched.Lock) + require.NoError(t, err) + assert.NotEqual(t, obsolete, installed) + data, err := os.ReadFile(sentinel) + require.NoError(t, err) + assert.Equal(t, "keep", string(data)) + other := fetched.Lock + other.Hash = migrationTestHash(t, map[string]string{"file.proto": "changed"}) + assert.NotEqual(t, installed, v1ModuleCachePath(cache.root, other)) + require.ErrorContains(t, cache.Install(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{other}}), "hash mismatch") + require.NoError(t, cache.VerifyCached(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{fetched.Lock}})) +} + +func TestSnapshotNeverRunsCheckoutFilters(t *testing.T) { + repository := t.TempDir() + files := map[string]string{"file.proto": "contents", ".gitattributes": "*.proto filter=blocker\n"} + for name, data := range files { + require.NoError(t, os.WriteFile(filepath.Join(repository, name), []byte(data), 0o644)) + } + runTestGit(t, repository, "init", "-q") + runTestGit(t, repository, "add", ".") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "initial") + global := filepath.Join(t.TempDir(), "gitconfig") + require.NoError(t, os.WriteFile(global, []byte("[filter \"blocker\"]\n smudge = false\n required = true\n"), 0o600)) + t.Setenv("GIT_CONFIG_GLOBAL", global) + cache := &Cache{root: t.TempDir()} + fetched, err := cache.Fetch(t.Context(), repository, "") + require.NoError(t, err) + assert.Equal(t, migrationTestHash(t, files), fetched.Lock.Hash) + require.NoError(t, cache.Install(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{fetched.Lock}})) +} + +func TestSnapshotHashIncludesMaterializedLogicalFileAlias(t *testing.T) { + t.Parallel() + repository := t.TempDir() + files := map[string]string{"protobuf.mod": "module " + repository + "\nroots proto\n", "physical/file.proto": "target bytes"} + for name, data := range files { + file := filepath.Join(repository, name) + require.NoError(t, os.MkdirAll(filepath.Dir(file), 0o755)) + require.NoError(t, os.WriteFile(file, []byte(data), 0o644)) + } + require.NoError(t, os.MkdirAll(filepath.Join(repository, "proto"), 0o755)) + require.NoError(t, os.Symlink("../physical/file.proto", filepath.Join(repository, "proto/alias.proto"))) + require.NoError(t, os.Symlink("missing", filepath.Join(repository, "proto/aaa_auxiliary"))) + runTestGit(t, repository, "init", "-q") + runTestGit(t, repository, "add", ".") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "initial") + cache := &Cache{root: t.TempDir()} + fetched, err := cache.Fetch(t.Context(), repository, "") + require.NoError(t, err) + files["proto/alias.proto"] = files["physical/file.proto"] + assert.Equal(t, migrationTestHash(t, files), fetched.Lock.Hash) + require.NoError(t, cache.Install(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{fetched.Lock}})) + installed, _, err := cache.Cached(fetched.Lock) + require.NoError(t, err) + actual, err := hashSnapshotV1Files(installed) + require.NoError(t, err) + assert.Equal(t, fetched.Lock.Hash, actual) +} + +func TestSnapshotNestedMetadataAliasFailurePreservesBoundary(t *testing.T) { + t.Parallel() + repository := snapshotPolicyFixture(t, map[string]string{"proto/child/item.proto": "syntax = \"proto3\";\n"}, map[string]string{"proto/child/protobuf.mod": "missing"}) + _, err := (&Cache{root: t.TempDir()}).Fetch(t.Context(), repository, "") + require.ErrorContains(t, err, "proto/child/protobuf.mod") +} + +func TestSnapshotCachedPolicyResolvesCustomAliases(t *testing.T) { + t.Parallel() + for _, tt := range []struct{ name, reference, link, target string }{ + {name: "file", reference: "./base.yaml", link: "base.yaml", target: "policies/strict.yaml"}, + {name: "directory namespace", reference: "./config/strict.yaml", link: "config", target: "policies"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + raw := []byte("version: v1\nlinters:\n extends: " + tt.reference + "\n") + repository := snapshotPolicyFixture(t, map[string]string{"easyp.yaml": string(raw), "policies/strict.yaml": "version: v1\nlinters:\n default: MINIMAL\n"}, map[string]string{tt.link: tt.target, "unused.yaml": "/outside/policy.yaml"}) + cache := &Cache{root: t.TempDir()} + fetched, err := cache.Fetch(t.Context(), repository, "") + require.NoError(t, err) + require.NoError(t, cache.Install(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{fetched.Lock}})) + installed, _, err := cache.Cached(fetched.Lock) + require.NoError(t, err) + cfg, err := v1.ParsePolicyLiteral(strings.NewReader(string(raw))) + require.NoError(t, err) + presence, err := v1.ParsePolicyPresence(raw) + require.NoError(t, err) + resolved, err := policyresolver.NewResolver(installed, nil).ResolveLint(t.Context(), policyresolver.LintInput{PolicyPath: filepath.Join(installed, "easyp.yaml"), Policy: cfg, Presence: presence, ModuleDir: installed}) + require.NoError(t, err) + require.Equal(t, "MINIMAL", resolved.Policy.Linters.Default) + info, err := os.Lstat(filepath.Join(installed, filepath.FromSlash(strings.TrimPrefix(tt.reference, "./")))) + require.NoError(t, err) + require.True(t, info.Mode().IsRegular()) + }) + } +} + +func TestSnapshotReferencedPolicyAliasMustResolve(t *testing.T) { + t.Parallel() + for _, tt := range []struct{ name, target string }{ + {name: "missing", target: "missing"}, + {name: "external", target: "../outside.yaml"}, + {name: "cycle", target: "base.yaml"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + repository := snapshotPolicyFixture(t, map[string]string{"easyp.yaml": "version: v1\nlinters:\n extends: ./base.yaml\n"}, map[string]string{"base.yaml": tt.target}) + cache := &Cache{root: t.TempDir()} + fetched, err := cache.Fetch(t.Context(), repository, "") + require.NoError(t, err) + require.NoError(t, cache.Install(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{fetched.Lock}})) + installed, _, err := cache.Cached(fetched.Lock) + require.NoError(t, err) + raw := []byte("version: v1\nlinters:\n extends: ./base.yaml\n") + policy, err := v1.ParsePolicyLiteral(strings.NewReader(string(raw))) + require.NoError(t, err) + _, err = policyresolver.NewResolver(installed, nil).ResolveLint(t.Context(), policyresolver.LintInput{PolicyPath: filepath.Join(installed, v1.PolicyFile), Policy: policy}) + require.Error(t, err) + }) + } +} + +func snapshotPolicyFixture(t *testing.T, files, links map[string]string) string { + t.Helper() + repository := t.TempDir() + files[v1.ModuleFile] = "module " + repository + "\nroots proto\n" + files["proto/main.proto"] = "syntax = \"proto3\";\n" + for name, data := range files { + target := filepath.Join(repository, filepath.FromSlash(name)) + require.NoError(t, os.MkdirAll(filepath.Dir(target), 0o755)) + require.NoError(t, os.WriteFile(target, []byte(data), 0o644)) + } + for name, target := range links { + link := filepath.Join(repository, filepath.FromSlash(name)) + require.NoError(t, os.MkdirAll(filepath.Dir(link), 0o755)) + require.NoError(t, os.Symlink(target, link)) + } + runTestGit(t, repository, "init", "-q") + runTestGit(t, repository, "add", ".") + runTestGit(t, repository, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "initial") + return repository +} + +func TestSnapshotRemoteCustomPolicyAliasesWithoutRootPolicy(t *testing.T) { + t.Parallel() + for _, tt := range []struct{ name, fragment, link, target string }{ + {name: "custom extension", fragment: "base.rules", link: "base.rules", target: "policies/strict.yaml"}, + {name: "directory namespace", fragment: "config/strict.yaml", link: "config", target: "policies"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + repository := snapshotPolicyFixture(t, map[string]string{"policies/strict.yaml": "version: v1\nlinters:\n default: MINIMAL\n"}, map[string]string{tt.link: tt.target, "unused.yaml": "missing"}) + cache := &Cache{root: t.TempDir()} + fetched, err := cache.Fetch(t.Context(), repository, "") + require.NoError(t, err) + require.NoError(t, cache.Install(t.Context(), v1.Lock{Version: 1, Modules: []v1.LockedModule{fetched.Lock}})) + installed, _, err := cache.Cached(fetched.Lock) + require.NoError(t, err) + consumer := t.TempDir() + raw := []byte("version: v1\nlinters:\n extends: " + repository + "#" + tt.fragment + "\n") + require.NoError(t, os.WriteFile(filepath.Join(consumer, v1.PolicyFile), raw, 0o644)) + cfg, err := v1.ParsePolicyLiteral(strings.NewReader(string(raw))) + require.NoError(t, err) + presence, err := v1.ParsePolicyPresence(raw) + require.NoError(t, err) + resolver := policyresolver.NewResolver(consumer, func(context.Context, string) (modules.PolicyGraph, error) { + return modules.PolicyGraph{repository: {Name: repository, Directory: installed}}, nil + }) + resolved, err := resolver.ResolveLint(t.Context(), policyresolver.LintInput{PolicyPath: filepath.Join(consumer, v1.PolicyFile), Policy: cfg, Presence: presence, ModuleDir: consumer}) + require.NoError(t, err) + require.Equal(t, "MINIMAL", resolved.Policy.Linters.Default) + }) + } +} + +func TestSnapshotUnusedNestedFailureHonorsAncestorModule(t *testing.T) { + t.Parallel() + repository := snapshotPolicyFixture(t, map[string]string{"proto/nested/protobuf.mod": "module example.com/nested\nroots .\n", "proto/nested/deep/item.proto": "syntax = \"proto3\";\n"}, map[string]string{"proto/nested/deep/protobuf.mod": "missing"}) + _, err := (&Cache{root: t.TempDir()}).Fetch(t.Context(), repository, "") + require.NoError(t, err) +} diff --git a/internal/adapters/gitsnapshot/destination.go b/internal/adapters/gitsnapshot/destination.go new file mode 100644 index 00000000..3f576a26 --- /dev/null +++ b/internal/adapters/gitsnapshot/destination.go @@ -0,0 +1,44 @@ +package gitsnapshot + +import ( + "errors" + "fmt" + "os" + "path/filepath" + "strings" +) + +// ErrPathCollision means distinct logical Git paths share one host destination. +var ErrPathCollision = errors.New("snapshot path collision") + +// ValidateDestination rejects distinct Git paths that the host filesystem aliases. +// Git paths stay case-sensitive even on a case-insensitive staging filesystem. +func ValidateDestination(directory, name string) error { + if name == "." || name == "" { + return nil + } + current := directory + for _, component := range strings.Split(filepath.FromSlash(name), string(filepath.Separator)) { + requested := filepath.Join(current, component) + _, err := os.Lstat(requested) + if errors.Is(err, os.ErrNotExist) { + return nil + } + if err != nil { + return fmt.Errorf("Lstat: %w", err) + } + entries, err := os.ReadDir(current) + if err != nil { + return fmt.Errorf("ReadDir: %w", err) + } + exact := false + for _, entry := range entries { + exact = exact || entry.Name() == component + } + if !exact { + return fmt.Errorf("%w: %q resolves to a different spelling below %q", ErrPathCollision, name, current) + } + current = requested + } + return nil +} diff --git a/internal/adapters/gitsnapshot/fs.go b/internal/adapters/gitsnapshot/fs.go new file mode 100644 index 00000000..a8b90e41 --- /dev/null +++ b/internal/adapters/gitsnapshot/fs.go @@ -0,0 +1,265 @@ +// Package gitsnapshot exposes one immutable Git commit as a filesystem. +package gitsnapshot + +import ( + "errors" + "fmt" + "io" + "io/fs" + "path" + "slices" + "strings" + "time" + + "github.com/go-git/go-git/v5" + "github.com/go-git/go-git/v5/plumbing" + "github.com/go-git/go-git/v5/plumbing/filemode" + "github.com/go-git/go-git/v5/plumbing/object" +) + +// ErrGitlink marks a tracked submodule boundary without reading its contents. +var ErrGitlink = errors.New("gitlink boundary") + +// FS reads tracked trees and blobs without consulting a checkout or host links. +// Symlinks are exposed to fs.ReadLinkFS; callers choose their bounded resolution policy. +type FS struct { + repo *git.Repository + tree *object.Tree + raw *repositoryFS +} + +// New opens the tree pinned by commit in repo. +func New(repo *git.Repository, commit plumbing.Hash) (*FS, error) { + revision, err := repo.CommitObject(commit) + if err != nil { + return nil, fmt.Errorf("CommitObject: %w", err) + } + tree, err := revision.Tree() + if err != nil { + return nil, fmt.Errorf("Tree: %w", err) + } + return &FS{repo: repo, tree: tree}, nil +} + +func (f *FS) lookup(name string) (object.TreeEntry, error) { + if !fs.ValidPath(name) { + return object.TreeEntry{}, fs.ErrInvalid + } + if name == "." { + return object.TreeEntry{Name: ".", Mode: filemode.Dir, Hash: f.tree.Hash}, nil + } + tree := f.tree + parts := strings.Split(name, "/") + for i, part := range parts { + var entry *object.TreeEntry + for j := range tree.Entries { + if tree.Entries[j].Name == part { + entry = &tree.Entries[j] + break + } + } + if entry == nil { + return object.TreeEntry{}, fs.ErrNotExist + } + if entry.Mode == filemode.Submodule { + return object.TreeEntry{}, fmt.Errorf("%w at %q", ErrGitlink, strings.Join(parts[:i+1], "/")) + } + if i == len(parts)-1 { + return *entry, nil + } + if entry.Mode != filemode.Dir { + return object.TreeEntry{}, fmt.Errorf("non-directory Git component %q", strings.Join(parts[:i+1], "/")) + } + child, err := f.repo.TreeObject(entry.Hash) + if err != nil { + return object.TreeEntry{}, fmt.Errorf("TreeObject: %w", err) + } + tree = child + } + return object.TreeEntry{}, fs.ErrNotExist +} + +func (f *FS) info(entry object.TreeEntry) (fileInfo, error) { + info := fileInfo{name: entry.Name} + switch entry.Mode { + case filemode.Dir: + info.mode = fs.ModeDir | 0o755 + case filemode.Regular, filemode.Deprecated: + info.mode = 0o644 + case filemode.Executable: + info.mode = 0o755 + case filemode.Symlink: + info.mode = fs.ModeSymlink | 0o777 + case filemode.Submodule: + return fileInfo{}, fmt.Errorf("%w at %q", ErrGitlink, entry.Name) + default: + return fileInfo{}, fmt.Errorf("unsupported Git mode %s at %q", entry.Mode, entry.Name) + } + if entry.Mode != filemode.Dir { + blob, err := f.repo.BlobObject(entry.Hash) + if err != nil { + return fileInfo{}, fmt.Errorf("BlobObject: %w", err) + } + info.size = blob.Size + } + return info, nil +} + +// Lstat describes a tracked node without following a symbolic link. +func (f *FS) Lstat(name string) (fs.FileInfo, error) { + if f.raw != nil { + return f.raw.Lstat(name) + } + entry, err := f.lookup(name) + if err != nil { + return nil, &fs.PathError{Op: "lstat", Path: name, Err: err} + } + info, err := f.info(entry) + if err != nil { + return nil, &fs.PathError{Op: "lstat", Path: name, Err: err} + } + return info, nil +} + +// ReadLink reads the committed pointer bytes of a tracked symbolic link. +func (f *FS) ReadLink(name string) (string, error) { + if f.raw != nil { + return f.raw.ReadLink(name) + } + entry, err := f.lookup(name) + if err != nil { + return "", &fs.PathError{Op: "readlink", Path: name, Err: err} + } + if entry.Mode != filemode.Symlink { + return "", &fs.PathError{Op: "readlink", Path: name, Err: fs.ErrInvalid} + } + file, err := f.Open(name) + if err != nil { + return "", err + } + data, err := io.ReadAll(file) + closeErr := file.Close() + if err != nil { + return "", fmt.Errorf("ReadAll: %w", err) + } + if closeErr != nil { + return "", fmt.Errorf("Close: %w", closeErr) + } + return string(data), nil +} + +// Open reads a committed blob or directory. It never follows links itself. +func (f *FS) Open(name string) (fs.File, error) { + if f.raw != nil { + return f.raw.Open(name) + } + entry, err := f.lookup(name) + if err != nil { + return nil, &fs.PathError{Op: "open", Path: name, Err: err} + } + info, err := f.info(entry) + if err != nil { + return nil, &fs.PathError{Op: "open", Path: name, Err: err} + } + if info.IsDir() { + entries, err := f.ReadDir(name) + if err != nil { + return nil, err + } + return &directory{info: info, entries: entries}, nil + } + blob, err := f.repo.BlobObject(entry.Hash) + if err != nil { + return nil, fmt.Errorf("BlobObject: %w", err) + } + reader, err := blob.Reader() + if err != nil { + return nil, fmt.Errorf("Reader: %w", err) + } + return &blobFile{ReadCloser: reader, info: info}, nil +} + +// ReadDir enumerates direct tracked children in lexical order. +func (f *FS) ReadDir(name string) ([]fs.DirEntry, error) { + if f.raw != nil { + return f.raw.ReadDir(name) + } + entry, err := f.lookup(name) + if err != nil { + return nil, &fs.PathError{Op: "readdir", Path: name, Err: err} + } + if entry.Mode != filemode.Dir { + return nil, &fs.PathError{Op: "readdir", Path: name, Err: fs.ErrInvalid} + } + tree, err := f.repo.TreeObject(entry.Hash) + if err != nil { + return nil, fmt.Errorf("TreeObject: %w", err) + } + entries := make([]fs.DirEntry, 0, len(tree.Entries)) + for _, child := range tree.Entries { + // Gitlinks are represented as inaccessible directories so bounded walkers + // can omit auxiliary submodules but fail when a selected path crosses one. + if child.Mode == filemode.Submodule { + entries = append(entries, gitlinkEntry{name: child.Name}) + continue + } + info, err := f.info(child) + if err != nil { + return nil, fmt.Errorf("info: %w", err) + } + entries = append(entries, fs.FileInfoToDirEntry(info)) + } + slices.SortFunc(entries, func(a, b fs.DirEntry) int { return strings.Compare(a.Name(), b.Name()) }) + return entries, nil +} + +type fileInfo struct { + name string + mode fs.FileMode + size int64 +} + +func (i fileInfo) Name() string { return path.Base(i.name) } +func (i fileInfo) Size() int64 { return i.size } +func (i fileInfo) Mode() fs.FileMode { return i.mode } +func (i fileInfo) ModTime() time.Time { return time.Time{} } +func (i fileInfo) IsDir() bool { return i.mode.IsDir() } +func (i fileInfo) Sys() any { return nil } + +type blobFile struct { + io.ReadCloser + info fileInfo +} + +func (f *blobFile) Stat() (fs.FileInfo, error) { return f.info, nil } + +type directory struct { + info fileInfo + entries []fs.DirEntry + offset int +} + +func (d *directory) Stat() (fs.FileInfo, error) { return d.info, nil } +func (d *directory) Close() error { return nil } +func (d *directory) Read([]byte) (int, error) { return 0, fs.ErrInvalid } +func (d *directory) ReadDir(n int) ([]fs.DirEntry, error) { + if d.offset >= len(d.entries) && n > 0 { + return nil, io.EOF + } + end := len(d.entries) + if n > 0 { + end = min(end, d.offset+n) + } + result := d.entries[d.offset:end] + d.offset = end + return result, nil +} + +type gitlinkEntry struct{ name string } + +func (e gitlinkEntry) Name() string { return e.name } +func (e gitlinkEntry) IsDir() bool { return true } +func (e gitlinkEntry) Type() fs.FileMode { return fs.ModeDir } +func (e gitlinkEntry) Info() (fs.FileInfo, error) { + return nil, fmt.Errorf("%w at %q", ErrGitlink, e.name) +} diff --git a/internal/adapters/gitsnapshot/fs_test.go b/internal/adapters/gitsnapshot/fs_test.go new file mode 100644 index 00000000..cce46bd8 --- /dev/null +++ b/internal/adapters/gitsnapshot/fs_test.go @@ -0,0 +1,76 @@ +package gitsnapshot + +import ( + "io" + "io/fs" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + "github.com/go-git/go-git/v5" + "github.com/go-git/go-git/v5/plumbing" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/easyp-tech/easyp/internal/sourceview" +) + +func TestPinnedTreeReadsAliasesWithoutHostTargets(t *testing.T) { + t.Parallel() + directory := t.TempDir() + require.NoError(t, os.MkdirAll(filepath.Join(directory, "physical"), 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(directory, "physical/file.proto"), []byte("pinned bytes"), 0o644)) + require.NoError(t, os.Symlink("physical", filepath.Join(directory, "logical"))) + runGit(t, directory, "init", "-q") + runGit(t, directory, "add", ".") + runGit(t, directory, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "initial") + commit := runGit(t, directory, "rev-parse", "HEAD") + repo, err := git.PlainOpen(directory) + require.NoError(t, err) + tree, err := New(repo, plumbing.NewHash(commit)) + require.NoError(t, err) + require.NoError(t, os.WriteFile(filepath.Join(directory, "physical/file.proto"), []byte("changed host bytes"), 0o644)) + pointer, err := fs.ReadLink(tree, "logical") + require.NoError(t, err) + assert.Equal(t, "physical", pointer) + info, err := fs.Lstat(tree, "logical") + require.NoError(t, err) + assert.Equal(t, fs.ModeSymlink, info.Mode()&fs.ModeSymlink) + file, err := sourceview.New(tree).Open(t.Context(), "logical/file.proto") + require.NoError(t, err) + data, err := io.ReadAll(file) + closeErr := file.Close() + require.NoError(t, err) + require.NoError(t, closeErr) + assert.Equal(t, "pinned bytes", string(data)) +} + +func TestTreeRejectsGitlinkCrossings(t *testing.T) { + t.Parallel() + directory := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(directory, "file"), []byte("file"), 0o644)) + runGit(t, directory, "init", "-q") + runGit(t, directory, "add", ".") + runGit(t, directory, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "initial") + commit := runGit(t, directory, "rev-parse", "HEAD") + runGit(t, directory, "update-index", "--add", "--cacheinfo", "160000", commit, "submodule") + runGit(t, directory, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "gitlink") + repo, err := git.PlainOpen(directory) + require.NoError(t, err) + tree, err := New(repo, plumbing.NewHash(runGit(t, directory, "rev-parse", "HEAD"))) + require.NoError(t, err) + _, err = fs.Lstat(tree, "submodule/file.proto") + require.ErrorContains(t, err, "gitlink boundary") + _, err = tree.Open("../file") + require.ErrorIs(t, err, fs.ErrInvalid) +} + +func runGit(t *testing.T, directory string, args ...string) string { + t.Helper() + command := exec.Command("git", append([]string{"-C", directory}, args...)...) + data, err := command.CombinedOutput() + require.NoError(t, err, string(data)) + return strings.TrimSpace(string(data)) +} diff --git a/internal/adapters/gitsnapshot/repository.go b/internal/adapters/gitsnapshot/repository.go new file mode 100644 index 00000000..4e5b0dd6 --- /dev/null +++ b/internal/adapters/gitsnapshot/repository.go @@ -0,0 +1,203 @@ +package gitsnapshot + +import ( + "bytes" + "context" + "encoding/hex" + "fmt" + "io" + "io/fs" + "os/exec" + "path" + "slices" + "strconv" + "strings" +) + +type repositoryEntry struct { + object string + info fileInfo + link bool +} + +type repositoryFS struct { + ctx context.Context + directory string + entries map[string]repositoryEntry +} + +// NewRepository reads an immutable tree through Git's format-aware object +// commands. In particular, SHA-256 object IDs must not become SHA-1 hashes. +func NewRepository(ctx context.Context, directory, commit string) (*FS, error) { + if !objectID(commit) { + return nil, fmt.Errorf("invalid pinned Git commit %q", commit) + } + backend := &repositoryFS{ctx: ctx, directory: directory, entries: map[string]repositoryEntry{ + ".": {info: fileInfo{name: ".", mode: fs.ModeDir | 0o755}}, + }} + raw, err := backend.command(nil, "ls-tree", "-rzt", "--full-tree", commit) + if err != nil { + return nil, fmt.Errorf("command: %w", err) + } + objects := make(map[string]bool) + for record := range strings.SplitSeq(string(raw), "\x00") { + if record == "" { + continue + } + metadata, name, ok := strings.Cut(record, "\t") + fields := strings.Fields(metadata) + if !ok || len(fields) != 3 || !fs.ValidPath(name) || !objectID(fields[2]) { + return nil, fmt.Errorf("invalid Git tree entry %q", record) + } + entry := repositoryEntry{object: fields[2], info: fileInfo{name: path.Base(name)}} + switch fields[0] { + case "040000": + entry.info.mode = fs.ModeDir | 0o755 + case "100644": + entry.info.mode = 0o644 + case "100755": + entry.info.mode = 0o755 + case "120000": + entry.info.mode = fs.ModeSymlink | 0o777 + case "160000": + entry.link = true + default: + return nil, fmt.Errorf("unsupported Git tree mode %q", fields[0]) + } + if fields[1] == "blob" { + objects[entry.object] = true + } + backend.entries[name] = entry + } + ids := make([]string, 0, len(objects)) + for id := range objects { + ids = append(ids, id) + } + slices.Sort(ids) + if len(ids) != 0 { + input := strings.NewReader(strings.Join(ids, "\n") + "\n") + checked, err := backend.command(input, "cat-file", "--batch-check") + if err != nil { + return nil, fmt.Errorf("command: %w", err) + } + sizes := make(map[string]int64, len(ids)) + for line := range strings.SplitSeq(strings.TrimSpace(string(checked)), "\n") { + fields := strings.Fields(line) + if len(fields) != 3 || fields[1] != "blob" { + return nil, fmt.Errorf("invalid Git blob metadata %q", line) + } + size, err := strconv.ParseInt(fields[2], 10, 64) + if err != nil || size < 0 { + return nil, fmt.Errorf("invalid Git blob size %q", line) + } + sizes[fields[0]] = size + } + for name, entry := range backend.entries { + entry.info.size = sizes[entry.object] + backend.entries[name] = entry + } + } + return &FS{raw: backend}, nil +} + +func objectID(value string) bool { + if len(value) != 40 && len(value) != 64 { + return false + } + _, err := hex.DecodeString(value) + return err == nil +} + +func (f *repositoryFS) command(stdin io.Reader, args ...string) ([]byte, error) { + cmd := exec.CommandContext(f.ctx, "git", append([]string{"-C", f.directory}, args...)...) + cmd.Stdin = stdin + raw, err := cmd.Output() + if err != nil { + if f.ctx.Err() != nil { + return nil, f.ctx.Err() + } + return nil, fmt.Errorf("Output: %w", err) + } + return raw, nil +} + +func (f *repositoryFS) lookup(name string) (repositoryEntry, error) { + if !fs.ValidPath(name) { + return repositoryEntry{}, fs.ErrInvalid + } + for prefix := name; prefix != "."; prefix = path.Dir(prefix) { + if entry, found := f.entries[prefix]; found && entry.link { + return repositoryEntry{}, fmt.Errorf("%w at %q", ErrGitlink, prefix) + } + } + entry, found := f.entries[name] + if !found { + return repositoryEntry{}, fs.ErrNotExist + } + return entry, nil +} + +func (f *repositoryFS) Lstat(name string) (fs.FileInfo, error) { + entry, err := f.lookup(name) + if err != nil { + return nil, &fs.PathError{Op: "lstat", Path: name, Err: err} + } + return entry.info, nil +} + +func (f *repositoryFS) ReadLink(name string) (string, error) { + entry, err := f.lookup(name) + if err != nil { + return "", &fs.PathError{Op: "readlink", Path: name, Err: err} + } + if entry.info.mode&fs.ModeSymlink == 0 { + return "", &fs.PathError{Op: "readlink", Path: name, Err: fs.ErrInvalid} + } + raw, err := f.command(nil, "cat-file", "blob", entry.object) + if err != nil { + return "", fmt.Errorf("command: %w", err) + } + return string(raw), nil +} + +func (f *repositoryFS) Open(name string) (fs.File, error) { + entry, err := f.lookup(name) + if err != nil { + return nil, &fs.PathError{Op: "open", Path: name, Err: err} + } + if entry.info.IsDir() { + children, err := f.ReadDir(name) + if err != nil { + return nil, fmt.Errorf("ReadDir: %w", err) + } + return &directory{info: entry.info, entries: children}, nil + } + raw, err := f.command(nil, "cat-file", "blob", entry.object) + if err != nil { + return nil, fmt.Errorf("command: %w", err) + } + return &blobFile{ReadCloser: io.NopCloser(bytes.NewReader(raw)), info: entry.info}, nil +} + +func (f *repositoryFS) ReadDir(name string) ([]fs.DirEntry, error) { + entry, err := f.lookup(name) + if err != nil || !entry.info.IsDir() { + if err == nil { + err = fs.ErrInvalid + } + return nil, &fs.PathError{Op: "readdir", Path: name, Err: err} + } + var children []fs.DirEntry + for child, entry := range f.entries { + if child == "." || path.Dir(child) != name { + continue + } + if entry.link { + children = append(children, gitlinkEntry{name: path.Base(child)}) + } else { + children = append(children, fs.FileInfoToDirEntry(entry.info)) + } + } + slices.SortFunc(children, func(a, b fs.DirEntry) int { return strings.Compare(a.Name(), b.Name()) }) + return children, nil +} diff --git a/internal/adapters/go_git/snapshot.go b/internal/adapters/go_git/snapshot.go index 4d553b53..260386b9 100644 --- a/internal/adapters/go_git/snapshot.go +++ b/internal/adapters/go_git/snapshot.go @@ -5,15 +5,19 @@ import ( "errors" "fmt" "io" + "io/fs" "os" "path" "path/filepath" + "strings" gogit "github.com/go-git/go-git/v5" - "github.com/go-git/go-git/v5/plumbing/filemode" - "github.com/go-git/go-git/v5/plumbing/object" + "github.com/easyp-tech/easyp/internal/adapters/gitsnapshot" + v1 "github.com/easyp-tech/easyp/internal/config/v1" "github.com/easyp-tech/easyp/internal/core" + "github.com/easyp-tech/easyp/internal/core/path_helpers" + "github.com/easyp-tech/easyp/internal/sourceview" ) // Snapshot contains revision-local protobuf sources and dependency metadata. @@ -61,9 +65,9 @@ func SnapshotRevision(ctx context.Context, directory, ref string) (_ *Snapshot, if err != nil { return nil, fmt.Errorf("baselineCommit: %w", err) } - tree, err := commit.Tree() + tree, err := gitsnapshot.New(repository, commit.Hash) if err != nil { - return nil, fmt.Errorf("Tree: %w", err) + return nil, fmt.Errorf("New: %w", err) } root, err := os.MkdirTemp("", "easyp-breaking-*") if err != nil { @@ -75,29 +79,27 @@ func SnapshotRevision(ctx context.Context, directory, ref string) (_ *Snapshot, _ = snapshot.Close() } }() - err = tree.Files().ForEach(func(file *object.File) error { - if err := ctx.Err(); err != nil { - return err + view := sourceview.New(tree) + err = view.Walk(ctx, ".", func(name string, resolved sourceview.Resolution, walkErr error) error { + if walkErr != nil { + if !snapshotInput(name) || snapshotAdditionalPolicy(name) { + return nil + } + return fmt.Errorf("baseline input alias %q: %w", name, walkErr) } - if !snapshotInput(file.Name) { + if resolved.Info.IsDir() { return nil } - relative := filepath.FromSlash(file.Name) - if !filepath.IsLocal(relative) { - return fmt.Errorf("invalid snapshot path %q", file.Name) + if !snapshotInput(name) { + return nil } - if file.Mode != filemode.Regular && file.Mode != filemode.Executable && file.Mode != filemode.Deprecated { - // Additional policy candidates are copied only when regular. An unused - // YAML symlink must not break an otherwise unrelated baseline check. If - // referenced as a policy, its absence is reported without following it. - if snapshotAdditionalPolicy(file.Name) { - return nil - } - return fmt.Errorf("baseline input %q is not a regular file", file.Name) + relative := filepath.FromSlash(name) + if !filepath.IsLocal(relative) { + return fmt.Errorf("invalid snapshot path %q", name) } - reader, err := file.Reader() + reader, err := view.Open(ctx, name) if err != nil { - return fmt.Errorf("Reader: %w", err) + return fmt.Errorf("Open: %w", err) } raw, readErr := io.ReadAll(reader) closeErr := reader.Close() @@ -107,6 +109,9 @@ func SnapshotRevision(ctx context.Context, directory, ref string) (_ *Snapshot, if closeErr != nil { return fmt.Errorf("Close: %w", closeErr) } + if err := gitsnapshot.ValidateDestination(root, name); err != nil { + return fmt.Errorf("ValidateDestination: %w", err) + } target := filepath.Join(root, relative) if err := os.MkdirAll(filepath.Dir(target), 0o700); err != nil { return fmt.Errorf("MkdirAll: %w", err) @@ -117,11 +122,82 @@ func SnapshotRevision(ctx context.Context, directory, ref string) (_ *Snapshot, return nil }) if err != nil { - return nil, fmt.Errorf("ForEach: %w", err) + return nil, fmt.Errorf("Walk: %w", err) + } + if err := validateSnapshotRoots(ctx, view, root); err != nil { + return nil, fmt.Errorf("validateSnapshotRoots: %w", err) } return snapshot, nil } +// Required roots come from revision-local manifests, not filename extensions. +// A failed extensionless root alias must not silently create an empty baseline. +func validateSnapshotRoots(ctx context.Context, view *sourceview.View, root string) error { + err := filepath.WalkDir(root, func(filename string, entry fs.DirEntry, walkErr error) error { + if walkErr != nil { + return walkErr + } + if entry.IsDir() || entry.Name() != v1.ModuleFile { + return nil + } + raw, err := os.ReadFile(filename) + if err != nil { + return fmt.Errorf("ReadFile: %w", err) + } + if !v1.IsModuleManifest(raw) { + return nil + } + module, err := v1.ParseModule(strings.NewReader(string(raw))) + if err != nil { + return fmt.Errorf("ParseModule: %w", err) + } + relative, err := filepath.Rel(root, filepath.Dir(filename)) + if err != nil { + return fmt.Errorf("Rel: %w", err) + } + for _, declared := range module.Roots { + logical := path.Join(filepath.ToSlash(relative), filepath.ToSlash(declared)) + resolved, err := view.Resolve(ctx, logical) + if err != nil { + if errors.Is(err, fs.ErrNotExist) && len(resolved.Links) == 0 { + continue + } + return fmt.Errorf("Resolve: baseline root %q: %w", logical, err) + } + if !resolved.Info.IsDir() { + return fmt.Errorf("baseline root %q is not a directory", logical) + } + if err := view.Walk(ctx, logical, func(name string, resolved sourceview.Resolution, walkErr error) error { + selectedRoot := filepath.Join(root, filepath.FromSlash(logical)) + selectedPath := filepath.Join(root, filepath.FromSlash(name)) + if path_helpers.HiddenOrVendorSourcePath(selectedRoot, selectedPath) || path_helpers.ShouldSkipV1SourceDir(selectedRoot, selectedPath) { + if resolved.Info != nil && resolved.Info.IsDir() { + return fs.SkipDir + } + return nil + } + if errors.Is(walkErr, gitsnapshot.ErrGitlink) { + if name != logical && len(resolved.Links) == 0 { + return nil + } + return walkErr + } + if walkErr != nil && errors.Is(walkErr, sourceview.ErrCycle) && resolved.Info != nil && resolved.Info.IsDir() { + return fmt.Errorf("baseline directory alias %q: %w", name, walkErr) + } + return nil + }); err != nil { + return fmt.Errorf("Walk: %w", err) + } + } + return nil + }) + if err != nil { + return fmt.Errorf("WalkDir: %w", err) + } + return nil +} + func snapshotInput(name string) bool { if extension := path.Ext(name); extension == ".proto" || extension == ".yaml" || extension == ".yml" { return true diff --git a/internal/adapters/go_git/snapshot_alias_test.go b/internal/adapters/go_git/snapshot_alias_test.go new file mode 100644 index 00000000..eef8ee85 --- /dev/null +++ b/internal/adapters/go_git/snapshot_alias_test.go @@ -0,0 +1,144 @@ +package go_git + +import ( + "os" + "path/filepath" + "testing" + + gogit "github.com/go-git/go-git/v5" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/easyp-tech/easyp/internal/adapters/gitsnapshot" +) + +func TestSnapshotRevisionMaterializesCommittedAliases(t *testing.T) { + t.Parallel() + root := t.TempDir() + for name, content := range map[string]string{ + "real/item.proto": "committed proto bytes\n", + "real/manifest.txt": "module example.test/app\nroots api\n", + "real/policy-source.txt": "version: v1\n", + } { + filename := filepath.Join(root, name) + require.NoError(t, os.MkdirAll(filepath.Dir(filename), 0o755)) + require.NoError(t, os.WriteFile(filename, []byte(content), 0o644)) + } + for name, target := range map[string]string{ + "alias.proto": "real/item.proto", + "api": "real", + "protobuf.mod": "real/manifest.txt", + "easyp.yaml": "real/policy-source.txt", + "README-link": "../outside", + } { + require.NoError(t, os.Symlink(target, filepath.Join(root, name))) + } + snapshotGit(t, root, "init", "-q", "-b", "baseline") + snapshotGit(t, root, "add", ".") + snapshotGit(t, root, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "aliases") + require.NoError(t, os.WriteFile(filepath.Join(root, "real/item.proto"), []byte("working tree changed\n"), 0o644)) + + baseline, err := SnapshotRevision(t.Context(), root, "baseline") + + require.NoError(t, err) + t.Cleanup(func() { assert.NoError(t, baseline.Close()) }) + for name, expected := range map[string]string{ + "alias.proto": "committed proto bytes\n", + "api/item.proto": "committed proto bytes\n", + "protobuf.mod": "module example.test/app\nroots api\n", + "easyp.yaml": "version: v1\n", + } { + filename := filepath.Join(baseline.Root, name) + actual, err := os.ReadFile(filename) + require.NoError(t, err) + assert.Equal(t, expected, string(actual)) + info, err := os.Lstat(filename) + require.NoError(t, err) + assert.True(t, info.Mode().IsRegular(), name) + } + assert.NoFileExists(t, filepath.Join(baseline.Root, "README-link")) +} + +func TestSnapshotRevisionRejectsRequiredUnsafeAliases(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name string + links map[string]string + manifest bool + }{ + {name: "external proto", links: map[string]string{"alias.proto": "../outside.proto"}}, + {name: "dangling metadata", links: map[string]string{"protobuf.mod": "missing"}}, + {name: "proto cycle", links: map[string]string{"alias.proto": "other.proto", "other.proto": "alias.proto"}}, + {name: "declared directory cycle", links: map[string]string{"proto": "."}, manifest: true}, + {name: "declared dangling root", links: map[string]string{"proto": "missing"}, manifest: true}, + {name: "declared external root", links: map[string]string{"proto": "../outside"}, manifest: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(root, "item.proto"), []byte("committed\n"), 0o644)) + if tt.manifest { + require.NoError(t, os.WriteFile(filepath.Join(root, "protobuf.mod"), []byte("module example.test/app\nroots proto\n"), 0o644)) + } + for name, target := range tt.links { + require.NoError(t, os.Symlink(target, filepath.Join(root, name))) + } + snapshotGit(t, root, "init", "-q", "-b", "baseline") + snapshotGit(t, root, "add", ".") + snapshotGit(t, root, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "unsafe aliases") + + baseline, err := SnapshotRevision(t.Context(), root, "baseline") + + require.Error(t, err) + assert.Nil(t, baseline) + }) + } +} + +func TestSnapshotRevisionIgnoresUnselectedDirectoryCycles(t *testing.T) { + t.Parallel() + for _, tt := range []struct{ name, path, marker string }{ + {name: "hidden", path: "proto/.aux/cycle"}, + {name: "vendor", path: "proto/easyp_vendor/cycle"}, + {name: "nested module", path: "proto/other/cycle", marker: "proto/other/protobuf.mod"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + require.NoError(t, os.MkdirAll(filepath.Dir(filepath.Join(root, tt.path)), 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(root, "protobuf.mod"), []byte("module example.test/app\nroots proto\n"), 0o644)) + require.NoError(t, os.WriteFile(filepath.Join(root, "proto/item.proto"), []byte("syntax = \"proto3\"; message Item {}\n"), 0o644)) + if tt.marker != "" { + require.NoError(t, os.WriteFile(filepath.Join(root, tt.marker), []byte("module example.test/other\nroots absent\n"), 0o644)) + } + require.NoError(t, os.Symlink("..", filepath.Join(root, tt.path))) + snapshotGit(t, root, "init", "-q", "-b", "baseline") + snapshotGit(t, root, "add", ".") + snapshotGit(t, root, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "unused directory cycles") + baseline, err := SnapshotRevision(t.Context(), root, "baseline") + require.NoError(t, err) + t.Cleanup(func() { assert.NoError(t, baseline.Close()) }) + assert.FileExists(t, filepath.Join(baseline.Root, "proto/item.proto")) + }) + } +} + +func TestSnapshotRevisionRejectsSelectedGitlink(t *testing.T) { + t.Parallel() + root := t.TempDir() + require.NoError(t, os.MkdirAll(filepath.Join(root, "proto"), 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(root, "protobuf.mod"), []byte("module example.test/app\nroots proto/submodule\n"), 0o644)) + require.NoError(t, os.WriteFile(filepath.Join(root, "proto/item.proto"), []byte("syntax = \"proto3\"; message Item {}\n"), 0o644)) + snapshotGit(t, root, "init", "-q", "-b", "baseline") + snapshotGit(t, root, "add", ".") + snapshotGit(t, root, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "sources") + repo, err := gogit.PlainOpen(root) + require.NoError(t, err) + head, err := repo.Head() + require.NoError(t, err) + snapshotGit(t, root, "update-index", "--add", "--cacheinfo", "160000,"+head.Hash().String()+",proto/submodule") + snapshotGit(t, root, "-c", "user.name=Test", "-c", "user.email=test@example.test", "commit", "-qm", "selected submodule") + baseline, err := SnapshotRevision(t.Context(), root, "baseline") + require.ErrorIs(t, err, gitsnapshot.ErrGitlink) + assert.Nil(t, baseline) +} diff --git a/internal/adapters/module_config/git_dependency.go b/internal/adapters/module_config/git_dependency.go index a28d94d6..c7d66170 100644 --- a/internal/adapters/module_config/git_dependency.go +++ b/internal/adapters/module_config/git_dependency.go @@ -190,6 +190,9 @@ func readGitDependencyManifest(path, source string) (v1.Module, bool, error) { // ReadGitDependencyAt verifies the candidate module directory before adapting repository-relative roots. func ReadGitDependencyAt(checkout, source, subdir string) (v1.Module, error) { + if err := validateGitDependencyIndex(checkout); err != nil { + return v1.Module{}, fmt.Errorf("validateGitDependencyIndex: %w", err) + } major, err := v1.ModulePathMajor(source) if err != nil { return v1.Module{}, fmt.Errorf("ModulePathMajor: %w", err) @@ -201,6 +204,9 @@ func ReadGitDependencyAt(checkout, source, subdir string) (v1.Module, error) { var selected *v1.Module var identityErr error for _, location := range locations { + if err := validateGitDependencyDirectory(checkout, location); err != nil { + return v1.Module{}, fmt.Errorf("validateGitDependencyDirectory: %w", err) + } manifestPath := filepath.Join(checkout, location, v1.ModuleFile) raw, err := readGitDependencyConfig(manifestPath) if errors.Is(err, os.ErrNotExist) { @@ -245,8 +251,8 @@ func ReadGitDependencyAt(checkout, source, subdir string) (v1.Module, error) { return v1.Module{}, fmt.Errorf("module %s requires a protobuf.mod declaring its exact identity in %v", source, locations) } -// A symlink would be omitted from the installed snapshot, so reading it here -// could give a downloaded module different roots or requirements from its cache. +// Metadata readers consume regular materialized snapshots. Pointer resolution +// belongs to the bounded source view before these format-specific readers run. func readGitDependencyConfig(path string) ([]byte, error) { info, err := os.Lstat(path) if err != nil { diff --git a/internal/adapters/module_config/git_dependency_buf.go b/internal/adapters/module_config/git_dependency_buf.go index 04d85b70..aafdd260 100644 --- a/internal/adapters/module_config/git_dependency_buf.go +++ b/internal/adapters/module_config/git_dependency_buf.go @@ -65,6 +65,9 @@ func readBufDependencyWorkspace(path string) (bufDependencyMetadata, error) { if !filepath.IsLocal(directory) && directory != "." { return bufDependencyMetadata{}, fmt.Errorf("%s: directory %q leaves the repository", path, directory) } + if err := validateGitDependencyDirectory(base, directory); err != nil { + return bufDependencyMetadata{}, fmt.Errorf("validateGitDependencyDirectory: %w", err) + } modulePath := filepath.Join(base, directory, bufModuleConfigFile) _, err := os.Lstat(modulePath) if os.IsNotExist(err) { diff --git a/internal/adapters/module_config/git_dependency_buf_test.go b/internal/adapters/module_config/git_dependency_buf_test.go index aef2feaa..1b068635 100644 --- a/internal/adapters/module_config/git_dependency_buf_test.go +++ b/internal/adapters/module_config/git_dependency_buf_test.go @@ -38,3 +38,40 @@ func TestReadBufDependencyModuleRejectsInvalidPaths(t *testing.T) { }) } } + +func TestReadBufDependencyWorkspaceRejectsSymlinkAncestorBeforeMetadataRead(t *testing.T) { + t.Parallel() + + for _, tt := range []struct { + name string + link string + root string + targetRoot string + }{ + {name: "directory_link", link: "proto", root: "proto", targetRoot: "."}, + {name: "ancestor_link", link: "api", root: "api/proto", targetRoot: "proto"}, + {name: "dangling_link", link: "proto", root: "proto", targetRoot: "missing"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + checkout, outside := t.TempDir(), t.TempDir() + if tt.name != "dangling_link" { + metadata := filepath.Join(outside, tt.targetRoot, bufModuleConfigFile) + require.NoError(t, os.MkdirAll(filepath.Dir(metadata), 0o755)) + require.NoError(t, os.WriteFile(metadata, []byte("version: v1\ndeps: [buf.build/acme/outside]\n"), 0o644)) + } + target := outside + if tt.name == "dangling_link" { + target = filepath.Join(outside, tt.targetRoot) + } + require.NoError(t, os.Symlink(target, filepath.Join(checkout, tt.link))) + workspace := filepath.Join(checkout, bufWorkConfigFile) + require.NoError(t, os.WriteFile(workspace, []byte("version: v1\ndirectories: ["+tt.root+"]\n"), 0o644)) + + _, err := readBufDependencyWorkspace(workspace) + + require.ErrorContains(t, err, "non-regular dependency directory") + require.NotContains(t, err.Error(), "buf.build/acme/outside") + }) + } +} diff --git a/internal/adapters/module_config/git_dependency_path.go b/internal/adapters/module_config/git_dependency_path.go new file mode 100644 index 00000000..867a648d --- /dev/null +++ b/internal/adapters/module_config/git_dependency_path.go @@ -0,0 +1,66 @@ +package moduleconfig + +import ( + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + + "github.com/easyp-tech/easyp/internal/adapters/gitindex" +) + +// Git can materialize a symlink as a regular file with core.symlinks=false. +// Check metadata modes before reading any of its bytes. Installed snapshots +// have no Git index and retain the filesystem checks in their config reader. +func validateGitDependencyIndex(checkout string) error { + _, err := os.Lstat(filepath.Join(checkout, ".git")) + if errors.Is(err, os.ErrNotExist) { + return nil + } + if err != nil { + return fmt.Errorf("Lstat: %w", err) + } + command := exec.Command("git", "-C", checkout, "ls-files", "--stage", "-z") + raw, err := command.Output() + if err != nil { + return fmt.Errorf("Output: %w", err) + } + entries, err := gitindex.Parse(string(raw)) + if err != nil { + return fmt.Errorf("Parse: %w", err) + } + for _, entry := range entries { + if IsGitDependencyConfigFile(filepath.Base(entry.Path)) && !entry.IsRegular() { + return fmt.Errorf("non-regular dependency config %q in Git index", entry.Path) + } + } + return nil +} + +// Walk one bounded directory path with Lstat before any child config lookup. +// Neither a symlink nor its materialized pointer file is a module directory. +func validateGitDependencyDirectory(checkout, directory string) error { + if directory == "" || directory == "." { + return nil + } + if !filepath.IsLocal(directory) { + return fmt.Errorf("dependency directory %q leaves the repository", directory) + } + current := checkout + for _, component := range strings.Split(filepath.Clean(directory), string(filepath.Separator)) { + current = filepath.Join(current, component) + info, err := os.Lstat(current) + if errors.Is(err, os.ErrNotExist) { + return nil + } + if err != nil { + return fmt.Errorf("Lstat: %w", err) + } + if !info.IsDir() { + return fmt.Errorf("non-regular dependency directory %q", current) + } + } + return nil +} diff --git a/internal/adapters/module_config/git_dependency_path_test.go b/internal/adapters/module_config/git_dependency_path_test.go new file mode 100644 index 00000000..00c88f31 --- /dev/null +++ b/internal/adapters/module_config/git_dependency_path_test.go @@ -0,0 +1,76 @@ +package moduleconfig + +import ( + "os" + "os/exec" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestReadGitDependencyAtRejectsMaterializedMetadataSymlink(t *testing.T) { + t.Parallel() + + for _, name := range []string{dependencyManifestFile, bufWorkConfigFile, bufModuleConfigFile, bufLockFile, legacyEasyPConfigFile} { + t.Run(name, func(t *testing.T) { + t.Parallel() + repository := t.TempDir() + // Reading this materialized pointer as YAML succeeds for easyp.yaml, + // even though the historical Git mode is not a regular config file. + require.NoError(t, os.Symlink("{}", filepath.Join(repository, name))) + dependencyPathTestGit(t, repository, "init", "-q") + dependencyPathTestGit(t, repository, "add", ".") + dependencyPathTestGit(t, repository, "-c", "user.name=EasyP Test", "-c", "user.email=test@example.com", "commit", "-qm", "metadata link") + checkout := t.TempDir() + dependencyPathTestGit(t, checkout, "clone", "--quiet", "--no-checkout", "--", repository, checkout) + dependencyPathTestGit(t, checkout, "-c", "core.symlinks=false", "checkout", "--quiet", "HEAD") + info, err := os.Lstat(filepath.Join(checkout, name)) + require.NoError(t, err) + require.True(t, info.Mode().IsRegular()) + + _, err = ReadGitDependencyAt(checkout, "example.com/dependency", "") + + require.ErrorContains(t, err, "non-regular dependency config") + }) + } +} + +func TestReadGitDependencyAtRejectsModuleDirectorySymlinkBeforeMetadataRead(t *testing.T) { + t.Parallel() + + for _, materialized := range []bool{false, true} { + name := "symlink" + if materialized { + name = "materialized" + } + t.Run(name, func(t *testing.T) { + t.Parallel() + repository, outside := t.TempDir(), t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(outside, dependencyManifestFile), []byte("module example.com/dependency\n"), 0o644)) + require.NoError(t, os.Symlink(outside, filepath.Join(repository, "api"))) + dependencyPathTestGit(t, repository, "init", "-q") + dependencyPathTestGit(t, repository, "add", ".") + dependencyPathTestGit(t, repository, "-c", "user.name=EasyP Test", "-c", "user.email=test@example.com", "commit", "-qm", "directory link") + checkout := t.TempDir() + dependencyPathTestGit(t, checkout, "clone", "--quiet", "--no-checkout", "--", repository, checkout) + if materialized { + dependencyPathTestGit(t, checkout, "-c", "core.symlinks=false", "checkout", "--quiet", "HEAD") + } else { + dependencyPathTestGit(t, checkout, "checkout", "--quiet", "HEAD") + } + + _, err := ReadGitDependencyAt(checkout, "example.com/dependency", "api") + + require.ErrorContains(t, err, "non-regular dependency directory") + }) + } +} + +func dependencyPathTestGit(t *testing.T, directory string, args ...string) { + t.Helper() + command := exec.CommandContext(t.Context(), "git", args...) + command.Dir = directory + output, err := command.CombinedOutput() + require.NoError(t, err, "git %v: %s", args, output) +} diff --git a/internal/adapters/module_config/local_dependency.go b/internal/adapters/module_config/local_dependency.go new file mode 100644 index 00000000..ba6242a1 --- /dev/null +++ b/internal/adapters/module_config/local_dependency.go @@ -0,0 +1,63 @@ +package moduleconfig + +import ( + "context" + "errors" + "fmt" + "io/fs" + "os" + "path/filepath" + + v1 "github.com/easyp-tech/easyp/internal/config/v1" + "github.com/easyp-tech/easyp/internal/sourceview" +) + +// ReadLocalDependency adapts local metadata through a bounded logical view. +// The regular temporary metadata tree keeps the Git compatibility reader's +// strict historical checkout policy independent of local alias support. +func ReadLocalDependency(directory, source string) (_ v1.Module, resultErr error) { + stage, err := os.MkdirTemp("", "easyp-local-metadata-*") + if err != nil { + return v1.Module{}, fmt.Errorf("MkdirTemp: %w", err) + } + defer func() { resultErr = errors.Join(resultErr, os.RemoveAll(stage)) }() + err = sourceview.WalkLocal(context.Background(), directory, ".", func(logical string, resolved sourceview.Resolution, walkErr error) error { + metadata := IsGitDependencyConfigFile(filepath.Base(logical)) + if walkErr != nil { + if metadata || logical == "." { + return walkErr + } + return nil + } + if resolved.Info.IsDir() { + if logical != "." && (filepath.Base(logical) == ".git" || filepath.Base(logical) == "easyp_vendor" || filepath.Base(logical) == "node_modules") { + return fs.SkipDir + } + err := os.MkdirAll(filepath.Join(stage, filepath.FromSlash(logical)), 0o755) + if err != nil { + return fmt.Errorf("MkdirAll: %w", err) + } + return nil + } + if !metadata { + return nil + } + raw, err := sourceview.ReadLocal(context.Background(), directory, logical) + if err != nil { + return fmt.Errorf("ReadLocal: %w", err) + } + err = os.WriteFile(filepath.Join(stage, filepath.FromSlash(logical)), raw, resolved.Info.Mode().Perm()) + if err != nil { + return fmt.Errorf("WriteFile: %w", err) + } + return nil + }) + if err != nil { + return v1.Module{}, fmt.Errorf("WalkLocal: %w", err) + } + module, err := ReadGitDependency(stage, source) + if err != nil { + return v1.Module{}, fmt.Errorf("ReadGitDependency: %w", err) + } + return module, nil +} diff --git a/internal/api/breaking_scope.go b/internal/api/breaking_scope.go index f6d28f2d..4950b8b1 100644 --- a/internal/api/breaking_scope.go +++ b/internal/api/breaking_scope.go @@ -2,6 +2,7 @@ package api import ( "context" + "errors" "fmt" "io" "io/fs" @@ -14,6 +15,8 @@ import ( "github.com/easyp-tech/easyp/internal/core/path_helpers" disk "github.com/easyp-tech/easyp/internal/fs/fs" "github.com/easyp-tech/easyp/internal/modules" + "github.com/easyp-tech/easyp/internal/sourceview" + "github.com/easyp-tech/easyp/internal/workspace" ) type breakingScope struct{ files []string } @@ -31,7 +34,10 @@ func discoverBreakingScopes(replacements *policyReplacementSources, scanRelative return nil, fmt.Errorf("Stat: %w", err) } manifests := make(map[string]v1.Module) - err := filepath.WalkDir(scan, func(path string, entry fs.DirEntry, walkErr error) error { + err := workspace.WalkAt(root, scan, func(path string, entry fs.DirEntry, walkErr error) error { + if errors.Is(walkErr, sourceview.ErrNestedRepository) && filepath.Ext(path) == ".proto" { + walkErr = nil + } if walkErr != nil { return walkErr } diff --git a/internal/api/breaking_v1.go b/internal/api/breaking_v1.go index 19c1f27f..9824c476 100644 --- a/internal/api/breaking_v1.go +++ b/internal/api/breaking_v1.go @@ -20,6 +20,7 @@ import ( "github.com/easyp-tech/easyp/internal/logger" "github.com/easyp-tech/easyp/internal/modules" policyresolver "github.com/easyp-tech/easyp/internal/policy" + "github.com/easyp-tech/easyp/internal/workspace" ) // checkV1Policies compares each module in an independent dependency context. @@ -142,13 +143,13 @@ func (b BreakingCheck) checkV1Policies(ctx *cli.Context, log logger.Logger, conf if err != nil { return nil, fmt.Errorf("baseline %s module %s: %w", cfg.AgainstGitRef, module, err) } - app := core.New(core.Options{Logger: log, ImportRoots: currentImports.Paths(), ImportFileAllowed: currentImports.FileAllowed(), BreakingCheckConfig: core.BreakingCheckConfig{ + app := core.New(core.Options{Logger: log, ImportRoots: currentImports.Paths(), ImportFileAllowed: currentImports.FileAllowed(), OpenSourceFile: currentImports.OpenSourceFile, BreakingCheckConfig: core.BreakingCheckConfig{ IgnoreDirs: append(append([]string(nil), ignorePaths...), defaultVendorDir), AgainstGitRef: cfg.AgainstGitRef, FilesCheck: slices.Contains(cfg.Use, core.BreakingCheckFilesCheck), Categories: cfg.Categories, IgnoreUnstable: cfg.IgnoreUnstable, }}) - found, err := app.CompareBreakingWithImports(ctx.Context, newBreakingWalker(repositoryRoot, currentFiles), newBreakingWalker(snapshot.Root, baselineFiles), baselineImports.Paths(), baselineImports.FileAllowed()) + found, err := app.CompareBreakingWithImports(ctx.Context, newBreakingWalker(repositoryRoot, currentFiles), newBreakingWalker(snapshot.Root, baselineFiles), core.BreakingImports{Paths: baselineImports.Paths(), FileAllowed: baselineImports.FileAllowed(), Open: baselineImports.OpenSourceFile}) if err != nil { return nil, fmt.Errorf("CompareBreaking module %s against %s: %w", module, cfg.AgainstGitRef, err) } @@ -215,7 +216,7 @@ func discoverV1BreakingPolicySources(replacements *policyReplacementSources, sca if missing { return []string{owner}, nil } - err = filepath.WalkDir(scanPath, func(path string, entry fs.DirEntry, walkErr error) error { + err = workspace.WalkAt(projectRoot, scanPath, func(path string, entry fs.DirEntry, walkErr error) error { if walkErr != nil { return walkErr } @@ -459,11 +460,11 @@ func (b BreakingCheck) checkV1ExtendedBreakingSource( if err != nil { return nil, fmt.Errorf("baseline %s module %s: %w", cfg.AgainstGitRef, module, err) } - app := core.New(core.Options{Logger: log, ImportRoots: currentImports.Paths(), ImportFileAllowed: currentImports.FileAllowed(), BreakingCheckConfig: core.BreakingCheckConfig{ + app := core.New(core.Options{Logger: log, ImportRoots: currentImports.Paths(), ImportFileAllowed: currentImports.FileAllowed(), OpenSourceFile: currentImports.OpenSourceFile, BreakingCheckConfig: core.BreakingCheckConfig{ IgnoreDirs: append(append([]string(nil), ignorePaths...), defaultVendorDir), AgainstGitRef: cfg.AgainstGitRef, FilesCheck: slices.Contains(cfg.Use, core.BreakingCheckFilesCheck), Categories: cfg.Categories, IgnoreUnstable: cfg.IgnoreUnstable, }}) - findings, err := app.CompareBreakingWithImports(ctx.Context, newBreakingWalker(repositoryRoot, currentFiles), newBreakingWalker(snapshot.Root, baselineFiles), baselineImports.Paths(), baselineImports.FileAllowed()) + findings, err := app.CompareBreakingWithImports(ctx.Context, newBreakingWalker(repositoryRoot, currentFiles), newBreakingWalker(snapshot.Root, baselineFiles), core.BreakingImports{Paths: baselineImports.Paths(), FileAllowed: baselineImports.FileAllowed(), Open: baselineImports.OpenSourceFile}) if err != nil { return nil, fmt.Errorf("CompareBreaking module %s against %s: %w", module, cfg.AgainstGitRef, err) } diff --git a/internal/api/frozen_scopes.go b/internal/api/frozen_scopes.go index 5bf69511..8ce7c0d5 100644 --- a/internal/api/frozen_scopes.go +++ b/internal/api/frozen_scopes.go @@ -7,6 +7,7 @@ import ( "path/filepath" v1 "github.com/easyp-tech/easyp/internal/config/v1" + "github.com/easyp-tech/easyp/internal/workspace" ) // selectedPolicyScopes retains explicit empty modules in frozen mode, so @@ -49,7 +50,7 @@ func selectedPolicyScopes(replacements *policyReplacementSources, relative strin } } if info.IsDir() { - err := filepath.WalkDir(scan, func(path string, entry fs.DirEntry, walkErr error) error { + err := workspace.WalkAt(root, scan, func(path string, entry fs.DirEntry, walkErr error) error { if walkErr != nil { return walkErr } diff --git a/internal/api/lint_v1.go b/internal/api/lint_v1.go index b096f11d..babe780d 100644 --- a/internal/api/lint_v1.go +++ b/internal/api/lint_v1.go @@ -3,6 +3,7 @@ package api import ( "bytes" "context" + "errors" "fmt" iofs "io/fs" "os" @@ -20,11 +21,12 @@ import ( "github.com/easyp-tech/easyp/internal/modules" policyresolver "github.com/easyp-tech/easyp/internal/policy" "github.com/easyp-tech/easyp/internal/rules" + "github.com/easyp-tech/easyp/internal/sourceview" "github.com/easyp-tech/easyp/internal/workspace" ) func (l Lint) actionV1(ctx *cli.Context, log logger.Logger, configPath, projectRoot, lintRoot string) error { - raw, err := os.ReadFile(configPath) + raw, err := workspace.ReadFile(projectRoot, configPath) if err != nil { return fmt.Errorf("ReadFile: %w", err) } @@ -34,7 +36,11 @@ func (l Lint) actionV1(ctx *cli.Context, log logger.Logger, configPath, projectR } searchDir := filepath.Join(lintRoot, ctx.String(flagLintDirectoryPath.Name)) var files []string - err = filepath.WalkDir(searchDir, func(path string, entry iofs.DirEntry, walkErr error) error { + err = workspace.WalkAt(projectRoot, searchDir, func(path string, entry iofs.DirEntry, walkErr error) error { + if errors.Is(walkErr, sourceview.ErrNestedRepository) && filepath.Ext(path) == ".proto" && !policySourceExcluded(projectRoot, path) { + files = append(files, path) + return nil + } if walkErr != nil { return walkErr } diff --git a/internal/api/ls_files_filters_test.go b/internal/api/ls_files_filters_test.go index 7c9f03a9..20a8c0bb 100644 --- a/internal/api/ls_files_filters_test.go +++ b/internal/api/ls_files_filters_test.go @@ -78,3 +78,20 @@ func TestListV1FilesRejectsLexicallyExcludedAlias(t *testing.T) { }) } } + +func TestListV1FilesAllowsDeclaredDependencyAliasAcrossNestedRepository(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "protobuf.mod", "module example.test/app\nroots .\nrequire example.test/dep\nreplace example.test/dep => ./dep\n") + writeV1GenerateFixture(t, root, "dep/.git", "gitdir: /unused\n") + writeV1GenerateFixture(t, root, "dep/buf.yaml", "version: v2\nmodules:\n - path: .\n includes: [selected]\n") + writeV1GenerateFixture(t, root, "dep/selected/model.proto", "syntax = \"proto3\";\n") + writeV1GenerateFixture(t, root, "client.proto", "syntax = \"proto3\"; import \"alias.proto\";\n") + require.NoError(t, os.Symlink("dep/selected/model.proto", filepath.Join(root, "alias.proto"))) + _, module, err := modules.ReadManifest(root) + require.NoError(t, err) + result, err := listV1Files(t.Context(), root, module, true, nil) + require.NoError(t, err) + require.Empty(t, result.Errors) + require.Contains(t, result.Files, v1ListedFile{AbsPath: filepath.ToSlash(filepath.Join(root, "alias.proto")), ImportPath: "alias.proto", Source: "workspace", Root: filepath.ToSlash(root)}) +} diff --git a/internal/api/ls_files_v1.go b/internal/api/ls_files_v1.go index 9be4dbb5..043492fd 100644 --- a/internal/api/ls_files_v1.go +++ b/internal/api/ls_files_v1.go @@ -16,6 +16,7 @@ import ( v1 "github.com/easyp-tech/easyp/internal/config/v1" "github.com/easyp-tech/easyp/internal/flags" "github.com/easyp-tech/easyp/internal/modules" + "github.com/easyp-tech/easyp/internal/workspace" "github.com/easyp-tech/easyp/wellknownimports" ) @@ -74,7 +75,7 @@ func (l LsFiles) Action(ctx *cli.Context) error { return fmt.Errorf("Getwd: %w", err) } if _, statErr := os.Stat(filepath.Join(root, v1.ModuleFile)); errors.Is(statErr, os.ErrNotExist) { - raw, readErr := os.ReadFile(filepath.Join(root, v1.PolicyFile)) + raw, readErr := workspace.ReadFile(root, filepath.Join(root, v1.PolicyFile)) if readErr == nil && v1.LegacyPolicy(raw) { return v1.ErrLegacyConfiguration } @@ -143,29 +144,30 @@ func listV1Files(ctx context.Context, moduleDir string, module v1.Module, includ return v1ListResult{}, fmt.Errorf("EnsureSources: %w", err) } } - allowed := append(append(modules.SourceRoots(nil), roots...), dependencies...).FileAllowed() + allSources := append(append(modules.SourceRoots(nil), roots...), dependencies...) for _, root := range roots { result.Roots = append(result.Roots, v1ListedRoot{Path: filepath.ToSlash(root.Path), Source: "workspace"}) - if err := indexV1ProtoRoot(root, "workspace", allowed, index, &result.Files); err != nil { + if err := indexV1ProtoRoot(root, "workspace", allSources, index, &result.Files); err != nil { return v1ListResult{}, fmt.Errorf("indexV1ProtoRoot: %w", err) } } if includeImports { for _, dependency := range dependencies { result.Roots = append(result.Roots, v1ListedRoot{Path: filepath.ToSlash(dependency.Path), Source: "dependency"}) - if err := indexV1ProtoRoot(dependency, "dependency", allowed, index, nil); err != nil { + if err := indexV1ProtoRoot(dependency, "dependency", allSources, index, nil); err != nil { return v1ListResult{}, fmt.Errorf("indexV1ProtoRoot: %w", err) } } - collectV1ListedImports(index, &result) + collectV1ListedImports(index, &result, allSources) } slices.SortFunc(result.Roots, func(a, b v1ListedRoot) int { return strings.Compare(a.Path, b.Path) }) slices.SortFunc(result.Files, func(a, b v1ListedFile) int { return strings.Compare(a.ImportPath, b.ImportPath) }) return result, nil } -func indexV1ProtoRoot(root modules.SourceRoot, source string, allowed func(string) bool, index map[string]v1ListedFile, selected *[]v1ListedFile) error { - return root.Walk(func(path string) error { +func indexV1ProtoRoot(root modules.SourceRoot, source string, sources modules.SourceRoots, index map[string]v1ListedFile, selected *[]v1ListedFile) error { + allowed := sources.FileAllowed() + return sources.WalkSelected(root, allowed, func(path string) error { if allowed != nil && !allowed(path) { return nil } @@ -186,7 +188,7 @@ func indexV1ProtoRoot(root modules.SourceRoot, source string, allowed func(strin }) } -func collectV1ListedImports(index map[string]v1ListedFile, result *v1ListResult) { +func collectV1ListedImports(index map[string]v1ListedFile, result *v1ListResult, sources modules.SourceRoots) { queue := append([]v1ListedFile(nil), result.Files...) seen := make(map[string]bool, len(queue)) for _, file := range queue { @@ -195,7 +197,7 @@ func collectV1ListedImports(index map[string]v1ListedFile, result *v1ListResult) for len(queue) > 0 { file := queue[0] queue = queue[1:] - imports, err := readV1ListedImports(file) + imports, err := readV1ListedImports(file, sources) if err != nil { result.Errors = append(result.Errors, v1ListError{Code: "parse_error", Message: fmt.Sprintf("%s: %v", file.ImportPath, err)}) continue @@ -235,9 +237,13 @@ func resolveV1ListedImport(owner, importPath string, index map[string]v1ListedFi return file, nil } -func readV1ListedImports(file v1ListedFile) ([]string, error) { +func readV1ListedImports(file v1ListedFile, sources modules.SourceRoots) ([]string, error) { if file.Source != "wellknown" { - return modules.ReadProtoImports(filepath.FromSlash(file.AbsPath)) + raw, err := sources.ReadSourceFile(filepath.FromSlash(file.AbsPath)) + if err != nil { + return nil, fmt.Errorf("ReadSourceFile: %w", err) + } + return modules.ParseProtoImports(file.ImportPath, raw) } raw, err := wellknownimports.Content.ReadFile(file.ImportPath) if err != nil { diff --git a/internal/api/migrate_interactive.go b/internal/api/migrate_interactive.go index f4bbfce0..00a744ed 100644 --- a/internal/api/migrate_interactive.go +++ b/internal/api/migrate_interactive.go @@ -13,6 +13,7 @@ import ( "github.com/easyp-tech/easyp/internal/adapters/gitmodules" v1 "github.com/easyp-tech/easyp/internal/config/v1" "github.com/easyp-tech/easyp/internal/migration" + "github.com/easyp-tech/easyp/internal/workspace" ) type migrationWizard struct { @@ -168,7 +169,7 @@ func (w migrationWizard) identity(ctx context.Context, root, value string, expli func migrationSuggestedIdentity(ctx context.Context, root string) string { var identity string - raw, err := os.ReadFile(filepath.Join(root, v1.ModuleFile)) + raw, err := workspace.ReadFile(root, filepath.Join(root, v1.ModuleFile)) if err == nil { module, err := v1.ParseModule(bytes.NewReader(raw)) if err == nil { diff --git a/internal/api/module_root.go b/internal/api/module_root.go index b9edeb71..f0454984 100644 --- a/internal/api/module_root.go +++ b/internal/api/module_root.go @@ -17,7 +17,7 @@ func moduleWorkingDir() (string, error) { if err != nil { policy, lookupErr := workspace.Policy(cwd) if lookupErr == nil { - raw, readErr := os.ReadFile(policy) + raw, readErr := workspace.ReadFileAt(policy) if readErr == nil && v1.LegacyPolicy(raw) { return "", v1.ErrLegacyConfiguration } diff --git a/internal/api/optional_file.go b/internal/api/optional_file.go index e30a2318..c7821f62 100644 --- a/internal/api/optional_file.go +++ b/internal/api/optional_file.go @@ -4,10 +4,12 @@ import ( "errors" "fmt" "os" + + "github.com/easyp-tech/easyp/internal/workspace" ) func readOptionalFile(path string) ([]byte, bool, error) { - raw, err := os.ReadFile(path) + raw, err := workspace.ReadFileAt(path) if errors.Is(err, os.ErrNotExist) { return nil, false, nil } diff --git a/internal/api/policy_imports.go b/internal/api/policy_imports.go index 54ab3546..334d23b8 100644 --- a/internal/api/policy_imports.go +++ b/internal/api/policy_imports.go @@ -5,6 +5,7 @@ import ( "context" "errors" "fmt" + "github.com/easyp-tech/easyp/internal/workspace" "os" "path/filepath" @@ -160,7 +161,7 @@ func (s *policyReplacementSources) replacementManifest(directory string) (v1.Mod if module, known := s.manifests[directory]; known { return module, nil } - raw, err := os.ReadFile(filepath.Join(directory, v1.ModuleFile)) + raw, err := workspace.ReadFileAt(filepath.Join(directory, v1.ModuleFile)) if err != nil && !errors.Is(err, os.ErrNotExist) { return v1.Module{}, fmt.Errorf("ReadFile: %w", err) } diff --git a/internal/api/runtime.go b/internal/api/runtime.go index c7e92d55..8e65073b 100644 --- a/internal/api/runtime.go +++ b/internal/api/runtime.go @@ -75,6 +75,7 @@ func buildCore(log logger.Logger, cfg config.Config, importRoots modules.SourceR Logger: log, ImportRoots: importRoots.Paths(), ImportFileAllowed: importRoots.FileAllowed(), + OpenSourceFile: importRoots.OpenSourceFile, CurrentProjectGitWalker: go_git.New(), BreakingCheckConfig: core.BreakingCheckConfig{ IgnoreDirs: append(append([]string(nil), cfg.BreakingCheck.Ignore...), defaultVendorDir), diff --git a/internal/config/v1/generate.go b/internal/config/v1/generate.go index d1142cb1..ef50f20e 100644 --- a/internal/config/v1/generate.go +++ b/internal/config/v1/generate.go @@ -20,10 +20,11 @@ type Generate struct { Options GenerateOptions `yaml:"options"` } -// GenerateTargets selects modules and their managed descriptor options. +// GenerateTargets selects module sources and their managed descriptor options. type GenerateTargets struct { - Modules []string `yaml:"modules"` + Modules []GenerateModule `yaml:"modules"` Packages []string `yaml:"packages"` + Paths []string `yaml:"paths"` Managed config.ManagedMode `yaml:"managed"` } @@ -32,6 +33,20 @@ type GenerateOptions struct { Go GoOptions `yaml:"go"` } +// HasSourceSelectors reports whether either the project or a selected module +// restricts generated sources. +func (g GenerateTargets) HasSourceSelectors() bool { + if len(g.Packages) > 0 || len(g.Paths) > 0 { + return true + } + for _, module := range g.Modules { + if module.HasSourceSelectors() { + return true + } + } + return false +} + // GoOptions controls the Go package prefix. A nil prefix allows inheritance. type GoOptions struct { PackagePrefix *string `yaml:"package_prefix"` @@ -58,9 +73,17 @@ func ParseGenerate(r io.Reader) (Generate, error) { if result.Version != "v1" { return Generate{}, fmt.Errorf("easyp.gen.yaml version must be v1") } + for i, module := range result.Generate.Modules { + if err := module.Validate(); err != nil { + return Generate{}, fmt.Errorf("generate.modules[%d]: %w", i, err) + } + } if err := ValidatePackageSelectors(result.Generate.Packages); err != nil { return Generate{}, err } + if err := ValidatePathSelectors(result.Generate.Paths); err != nil { + return Generate{}, err + } if err := result.Generate.Managed.Validate(); err != nil { return Generate{}, fmt.Errorf("generate.managed: %w", err) } diff --git a/internal/config/v1/generate_module.go b/internal/config/v1/generate_module.go new file mode 100644 index 00000000..85900986 --- /dev/null +++ b/internal/config/v1/generate_module.go @@ -0,0 +1,66 @@ +package v1 + +import ( + "fmt" + "strings" + + "gopkg.in/yaml.v3" +) + +// GenerateModule selects a module and optionally limits its generated sources. +// A string is shorthand for a module without paths or package filters. +type GenerateModule struct { + Module string `yaml:"module"` + Paths []string `yaml:"paths,omitempty"` + Packages []string `yaml:"packages,omitempty"` +} + +// UnmarshalYAML accepts a module name or an object containing its selectors. +func (m *GenerateModule) UnmarshalYAML(node *yaml.Node) error { + switch node.Kind { + case yaml.ScalarNode: + if node.Tag != "!!str" { + return fmt.Errorf("generate.modules entries must be strings or module objects") + } + *m = GenerateModule{Module: node.Value} + case yaml.MappingNode: + for i := 0; i < len(node.Content); i += 2 { + name := node.Content[i].Value + if name != "module" && name != "paths" && name != "packages" { + return fmt.Errorf("unknown generate.modules field %q", name) + } + } + type plain GenerateModule + if err := node.Decode((*plain)(m)); err != nil { + return fmt.Errorf("Decode: %w", err) + } + default: + return fmt.Errorf("generate.modules entries must be strings or module objects") + } + return nil +} + +// MarshalYAML keeps the unfiltered shorthand compact. +func (m GenerateModule) MarshalYAML() (any, error) { + if !m.HasSourceSelectors() { + return m.Module, nil + } + type plain GenerateModule + return plain(m), nil +} + +// HasSourceSelectors reports whether this module restricts generated sources. +func (m GenerateModule) HasSourceSelectors() bool { + return len(m.Paths) > 0 || len(m.Packages) > 0 +} + +// Validate checks the selected module name and optional source selectors. +func (m GenerateModule) Validate() error { + if strings.TrimSpace(m.Module) == "" { + return fmt.Errorf("module must not be empty") + } + if err := ValidatePackageSelectors(m.Packages); err != nil { + return err + } + return ValidatePathSelectors(m.Paths) +} diff --git a/internal/config/v1/generate_test.go b/internal/config/v1/generate_test.go index 90079e7f..67460656 100644 --- a/internal/config/v1/generate_test.go +++ b/internal/config/v1/generate_test.go @@ -21,7 +21,7 @@ options: if err != nil { t.Fatal(err) } - if got.Generate.Modules[0] != "proto/user" || got.Plugins[0].Name != "go" || got.Options.Go.PackagePrefix == nil || *got.Options.Go.PackagePrefix != "github.com/acme/gen/go" { + if got.Generate.Modules[0].Module != "proto/user" || got.Plugins[0].Name != "go" || got.Options.Go.PackagePrefix == nil || *got.Options.Go.PackagePrefix != "github.com/acme/gen/go" { t.Fatalf("unexpected generator: %#v", got) } } diff --git a/internal/config/v1/module_selection_test.go b/internal/config/v1/module_selection_test.go new file mode 100644 index 00000000..8d2d17fe --- /dev/null +++ b/internal/config/v1/module_selection_test.go @@ -0,0 +1,45 @@ +package v1 + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "gopkg.in/yaml.v3" +) + +func TestParseGenerateModuleSelectors(t *testing.T) { + t.Parallel() + gen, err := ParseGenerate(strings.NewReader("generate:\n modules:\n - proto/user\n - module: example.com/contracts\n paths: [proto/api]\n packages: [api.v1]\n paths: [.]\n packages: [api.v1]\n")) + require.NoError(t, err) + require.Len(t, gen.Generate.Modules, 2) + raw, err := yaml.Marshal(gen.Generate.Modules[1]) + require.NoError(t, err) + var module map[string]any + require.NoError(t, yaml.Unmarshal(raw, &module)) + assert.Equal(t, "example.com/contracts", module["module"]) + assert.Equal(t, []any{"proto/api"}, module["paths"]) + assert.Equal(t, []any{"api.v1"}, module["packages"]) + assert.Equal(t, []string{"."}, gen.Generate.Paths) + assert.Equal(t, []string{"api.v1"}, gen.Generate.Packages) +} + +func TestParseGenerateRejectsInvalidModuleSelectors(t *testing.T) { + t.Parallel() + for _, tt := range []struct{ name, entry string }{ + {name: "empty_module", entry: "{module: ''}"}, + {name: "missing_module", entry: "{paths: [proto]}"}, + {name: "unknown_field", entry: "{module: proto, path: proto/api}"}, + {name: "invalid_path", entry: "{module: proto, paths: [../outside]}"}, + {name: "invalid_package", entry: "{module: proto, packages: [api/*]}"}, + {name: "numeric_module", entry: "42"}, + {name: "boolean_module", entry: "{module: true}"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + _, err := ParseGenerate(strings.NewReader("generate:\n modules: [" + tt.entry + "]\n")) + require.Error(t, err) + }) + } +} diff --git a/internal/config/v1/paths.go b/internal/config/v1/paths.go new file mode 100644 index 00000000..fc427b7c --- /dev/null +++ b/internal/config/v1/paths.go @@ -0,0 +1,42 @@ +package v1 + +import ( + "fmt" + "regexp" + "strings" + "unicode/utf8" +) + +// Exclude portable path separators, glob characters, control characters and +// Unicode whitespace. Literal Unicode characters keep the schema pattern +// compatible with both Go and JSON Schema regular expressions. +const pathSelectorExcluded = `\\:*?\[\]"<>|\x00-\x20\x7f-\x9f` + "\u00a0\u1680\u2000-\u200a\u2028\u2029\u202f\u205f\u3000\ufeff" + +const pathSelectorCharacter = `[^/` + pathSelectorExcluded + `]` +const pathSelectorNonDotCharacter = `[^/.` + pathSelectorExcluded + `]` +const pathSelectorForbiddenPattern = `[` + pathSelectorExcluded + `]` + +// A segment can start with dots but cannot be exactly "." or "..". +const pathSelectorSegment = `(` + pathSelectorNonDotCharacter + `|\.` + pathSelectorNonDotCharacter + `|\.\.` + pathSelectorCharacter + `)` + pathSelectorCharacter + `*` + +// PathSelectorPattern is the grammar of a canonical, portable module-relative +// file or subtree selector. A single dot selects the whole source namespace. +const PathSelectorPattern = `^(\.|` + pathSelectorSegment + `(/` + pathSelectorSegment + `)*)$` + +var pathSelector = regexp.MustCompile(PathSelectorPattern) + +// ValidatePathSelectors checks literal path selection without inspecting sources. +func ValidatePathSelectors(paths []string) error { + for index, name := range paths { + if !utf8.ValidString(name) || !pathSelector.MatchString(name) { + return fmt.Errorf("generate.paths[%d]: %q is not a canonical portable module-relative file or directory path", index, name) + } + } + return nil +} + +// PathSelectorMatches reports whether a module-relative source path equals a validated +// selector or lies in its subtree. Matching is bounded by path components. +func PathSelectorMatches(selector, name string) bool { + return selector == "." || name == selector || strings.HasPrefix(name, selector+"/") +} diff --git a/internal/config/v1/paths_test.go b/internal/config/v1/paths_test.go new file mode 100644 index 00000000..929e6a2a --- /dev/null +++ b/internal/config/v1/paths_test.go @@ -0,0 +1,70 @@ +package v1 + +import ( + "strconv" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/easyp-tech/easyp/internal/config" +) + +func TestGeneratePathSelectorValidation(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name string + valid bool + }{ + {name: ".", valid: true}, {name: "mcp", valid: true}, + {name: "mcp/options/v1/options.proto", valid: true}, {name: ".sources/api.proto", valid: true}, + {name: "api.proto/messages.proto", valid: true}, {name: "api-v1/sub_dir", valid: true}, + {name: "..sources/api.proto", valid: true}, {name: ".../api.proto", valid: true}, + {name: ""}, {name: "./mcp"}, {name: "../mcp"}, {name: "mcp/../other"}, + {name: "mcp/./options"}, {name: "mcp/"}, {name: "mcp//options"}, {name: "/mcp"}, + {name: "C:/mcp"}, {name: `mcp\options`}, {name: "mcp/*"}, {name: "mcp/**"}, + {name: "mcp[ab]"}, {name: "mcp?"}, {name: "mcp options"}, {name: "mcp\noptions"}, + {name: "mcp\toptions"}, {name: "mcp\u00a0options"}, {name: "mcp\u2003options"}, + {name: "mcp\u2028options"}, {name: "mcp\x00options"}, {name: "mcp|options"}, + {name: "mcp\n"}, {name: "mcp\r"}, {name: "mcp\u2028"}, {name: "mcp\ufeff"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + raw := "version: v1\ngenerate:\n paths: [" + strconv.Quote(tt.name) + "]\nplugins: []\n" + _, err := ParseGenerate(strings.NewReader(raw)) + assert.Equal(t, !tt.valid, config.HasErrors(ValidateGenerateYAML([]byte(raw)))) + if tt.valid { + require.NoError(t, err) + return + } + require.ErrorContains(t, err, "generate.paths") + }) + } + require.NoError(t, ValidatePathSelectors(nil)) + require.NoError(t, ValidatePathSelectors([]string{"mcp", "mcp"})) +} + +func TestPathSelectorMatches(t *testing.T) { + t.Parallel() + tests := []struct { + name string + selector string + file string + want bool + }{ + {name: "all", selector: ".", file: "mcp/options.proto", want: true}, + {name: "exact_file", selector: "mcp/options.proto", file: "mcp/options.proto", want: true}, + {name: "subtree", selector: "mcp", file: "mcp/options/v1/options.proto", want: true}, + {name: "directory_with_proto_suffix", selector: "api.proto", file: "api.proto/messages.proto", want: true}, + {name: "prefix_lookalike", selector: "mcp", file: "mcp-copy/options.proto"}, + {name: "file_prefix_lookalike", selector: "mcp/options.proto", file: "mcp/options.proto-copy"}, + {name: "different_root", selector: "mcp", file: "examples/mcp/options.proto"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + assert.Equal(t, tt.want, PathSelectorMatches(tt.selector, tt.file)) + }) + } +} diff --git a/internal/config/v1/schema.go b/internal/config/v1/schema.go index c98f163d..f386267a 100644 --- a/internal/config/v1/schema.go +++ b/internal/config/v1/schema.go @@ -96,9 +96,26 @@ func documents() map[string]*schema { generate := fromType(reflect.TypeFor[Generate]()) generate.Properties["version"] = &schema{Type: "string", Const: "v1"} - generate.Properties["generate"].Properties["packages"].Description = "Exact protobuf packages among selected modules. Empty means all source files. Imports remain available for compilation; with_imports separately controls dependency generation." + module := fromType(reflect.TypeFor[GenerateModule]()) + module.Required = []string{"module"} + module.Properties["module"].MinLength = 1 + module.Properties["module"].Description = "Module identity, workspace-relative module directory, or declared dependency selected through the consumer manifest and lock." + module.Properties["packages"].Description = "Exact protobuf packages generated from this module, intersected with its paths and the project's global paths/packages. Omitted or empty means all packages. Each selector must match a resulting source in this module." + module.Properties["packages"].Items.Pattern = PackageSelectorPattern + pathDescription := "Literal files or directory subtrees relative to the selected module directory, not its protobuf import roots or the generator file. Empty or omitted means all module sources; . selects the whole module. Paths are intersected with generate.packages and module-local filters, must be canonical portable relative names, and never change import roots or required imports. Distinct from plugins.opts.paths, which controls plugin output layout." + module.Properties["paths"].Description = pathDescription + " Each selector must match a resulting source in this module." + module.Properties["paths"].Items.Pattern = PathSelectorPattern + module.Properties["paths"].Items.Not = &schema{Pattern: pathSelectorForbiddenPattern} + generate.Properties["generate"].Properties["modules"].Items = &schema{OneOf: []*schema{{Type: "string", MinLength: 1}, module}} + generate.Properties["generate"].Properties["modules"].Description = "Modules selected for generation. A string selects all sources; a module object adds its own paths/packages, intersected with global filters. Omitted selects the module containing this generator. Identical repeated selections are idempotent; conflicting filters for one module are rejected." + generate.Properties["generate"].Properties["packages"].Description = "Exact protobuf packages among selected modules, intersected with global paths and each module's paths/packages. Omitted or empty means all packages. Each selector must match a resulting source in at least one selected module. Imports remain available for compilation; with_imports separately controls dependency generation." generate.Properties["generate"].Properties["packages"].Items.Pattern = PackageSelectorPattern - generate.Properties["options"].Properties["go"].Properties["package_prefix"].Description = "Sets go_package without enabling managed defaults for other languages; full managed mode requires generate.managed.enabled." + generate.Properties["generate"].Properties["paths"].Description = pathDescription + " Each selector must match a resulting source across selected modules; module-local filters apply too. These filters can be used without generate.modules." + generate.Properties["generate"].Properties["paths"].Items.Pattern = PathSelectorPattern + // Some schema validators allow a final newline before the regex end anchor. + // Reject forbidden characters independently so they agree with CLI parsing. + generate.Properties["generate"].Properties["paths"].Items.Not = &schema{Pattern: pathSelectorForbiddenPattern} + generate.Properties["options"].Properties["go"].Properties["package_prefix"].Description = "Optional Go prefix. Omitted allows inheritance; explicit empty blocks prefix inheritance. A nonempty value sets go_package without enabling managed defaults for other languages; full managed mode requires generate.managed.enabled. Migration does not create this setting." plugin := generate.Properties["plugins"].Items plugin.Required = []string{"out"} plugin.Properties["with_imports"].Description = "Generate code for transitive imports with this plugin only; defaults to false. Independent from descriptor --include_imports." diff --git a/internal/core/breaking_alias_test.go b/internal/core/breaking_alias_test.go new file mode 100644 index 00000000..7dcea61c --- /dev/null +++ b/internal/core/breaking_alias_test.go @@ -0,0 +1,113 @@ +package core + +import ( + "io" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + diskfs "github.com/easyp-tech/easyp/internal/fs/fs" + "github.com/easyp-tech/easyp/internal/logger" + "github.com/easyp-tech/easyp/internal/sourceview" +) + +type aliasBreakingWalker struct { + DirWalker + root string + paths []string +} + +func TestBreakingUsesEachRevisionsSourceOpener(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name string + categories []string + }{ + {name: "parser"}, + {name: "descriptors", categories: []string{"FILE"}}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + current, baseline := t.TempDir(), t.TempDir() + currentDependency, baselineDependency := t.TempDir(), t.TempDir() + for _, root := range []string{current, baseline} { + require.NoError(t, os.WriteFile(filepath.Join(root, "app.proto"), []byte("syntax = \"proto3\"; package app.v1; import \"dep.proto\"; message App { dep.v1.Item item = 1; }"), 0o644)) + } + require.NoError(t, os.WriteFile(filepath.Join(currentDependency, "dep.proto"), []byte("syntax = \"proto3\"; package dep.v1; message Item { int32 value = 1; }"), 0o644)) + require.NoError(t, os.WriteFile(filepath.Join(baselineDependency, "dep.proto"), []byte("syntax = \"proto3\"; package dep.v1; message Item { string value = 1; }"), 0o644)) + opener := func(roots []string) func(string) (io.ReadCloser, error) { + return func(filename string) (io.ReadCloser, error) { + for _, root := range roots { + relative, err := filepath.Rel(root, filename) + if err == nil && filepath.IsLocal(relative) { + return sourceview.OpenLocal(t.Context(), root, relative) + } + } + return nil, &os.PathError{Op: "open", Path: filename, Err: os.ErrNotExist} + } + } + app := New(Options{Logger: logger.NewNop(), ImportRoots: []string{current, currentDependency}, OpenSourceFile: opener([]string{current, currentDependency}), BreakingCheckConfig: BreakingCheckConfig{Categories: tt.categories}}) + + issues, err := app.CompareBreakingWithImports(t.Context(), diskfs.NewFSWalker(current, "."), diskfs.NewFSWalker(baseline, "."), BreakingImports{Paths: []string{baseline, baselineDependency}, Open: opener([]string{baseline, baselineDependency})}) + + require.NoError(t, err) + assert.NotEmpty(t, issues, "changed imported contract must use the historical source opener") + }) + } +} + +func (w aliasBreakingWalker) RootPath() string { return w.root } + +func (w aliasBreakingWalker) WalkDir(visit func(string, error) error) error { + for _, path := range w.paths { + if err := visit(path, nil); err != nil { + return err + } + } + return nil +} + +func TestBreakingAliasesKeepLogicalFileIdentity(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name string + paths []string + message bool + }{ + {name: "different import names", paths: []string{"real.proto", "alias.proto"}}, + {name: "repeated same import", paths: []string{"alias.proto", "alias.proto"}}, + {name: "duplicate declarations target first", paths: []string{"real.proto", "alias.proto"}, message: true}, + {name: "duplicate declarations alias first", paths: []string{"alias.proto", "real.proto"}, message: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + source := "syntax = \"proto3\"; package alias.v1;\n" + if tt.message { + source += "message Item {}\n" + } + require.NoError(t, os.WriteFile(filepath.Join(root, "real.proto"), []byte(source), 0o644)) + require.NoError(t, os.Symlink("real.proto", filepath.Join(root, "alias.proto"))) + app := New(Options{Logger: logger.NewNop(), ImportRoots: []string{root}}) + + graph, err := app.compileBreakingGraph(t.Context(), aliasBreakingWalker{DirWalker: diskfs.NewFSWalker(root, "."), root: root, paths: tt.paths}) + + if tt.message { + require.ErrorContains(t, err, "already defined") + assert.Nil(t, graph) + return + } + require.NoError(t, err) + assert.Contains(t, graph.targets, "alias.proto") + if tt.name == "different import names" { + assert.Len(t, graph.targets, 2) + assert.Contains(t, graph.targets, "real.proto") + } else { + assert.Len(t, graph.targets, 1) + } + }) + } +} diff --git a/internal/core/breaking_check.go b/internal/core/breaking_check.go index 1b802538..985edb0f 100644 --- a/internal/core/breaking_check.go +++ b/internal/core/breaking_check.go @@ -3,6 +3,7 @@ package core import ( "context" "fmt" + "io" "log/slog" "maps" "path/filepath" @@ -53,14 +54,22 @@ func (c *Core) BreakingCheck(ctx context.Context, projectRoot, workingDir, path // CompareBreaking compares explicitly scoped inputs with revision-specific imports. // Neither filesystem can fall back to the other revision's dependency roots. func (c *Core) CompareBreaking(ctx context.Context, current, against DirWalker, againstImportRoots []string) ([]IssueInfo, error) { - return c.CompareBreakingWithImports(ctx, current, against, againstImportRoots, nil) + return c.CompareBreakingWithImports(ctx, current, against, BreakingImports{Paths: againstImportRoots}) } -// CompareBreakingWithImports keeps source filters isolated between the two revisions. -func (c *Core) CompareBreakingWithImports(ctx context.Context, current, against DirWalker, againstImportRoots []string, againstFileAllowed func(string) bool) ([]IssueInfo, error) { +// BreakingImports binds one revision's import paths to its filter and opener. +type BreakingImports struct { + Paths []string + FileAllowed func(string) bool + Open func(string) (io.ReadCloser, error) +} + +// CompareBreakingWithImports keeps source access isolated between revisions. +func (c *Core) CompareBreakingWithImports(ctx context.Context, current, against DirWalker, imports BreakingImports) ([]IssueInfo, error) { baseline := *c - baseline.importRoots = againstImportRoots - baseline.importFileAllowed = againstFileAllowed + baseline.importRoots = imports.Paths + baseline.importFileAllowed = imports.FileAllowed + baseline.sourceFileOpen = imports.Open if len(c.breakingCheckConfig.Categories) > 0 { profiles, err := selectedBreakingProfiles(c.breakingCheckConfig.Categories) if err != nil { diff --git a/internal/core/breaking_profiles.go b/internal/core/breaking_profiles.go index 3ad9b254..5d199d64 100644 --- a/internal/core/breaking_profiles.go +++ b/internal/core/breaking_profiles.go @@ -2,7 +2,10 @@ package core import ( "context" + "errors" "fmt" + "io" + "os" "path/filepath" "slices" "strings" @@ -17,6 +20,7 @@ import ( "github.com/easyp-tech/easyp/internal/config" "github.com/easyp-tech/easyp/internal/core/path_helpers" + "github.com/easyp-tech/easyp/internal/sourceview" ) type breakingGraph struct { @@ -81,7 +85,7 @@ func (c *Core) compileBreakingGraph(ctx context.Context, walker DirWalker) (*bre root := rooted.RootPath() targets := make([]string, 0) targetPaths := make(map[string]string) - physical := make(map[string]bool) + seen := make(map[string]bool) err := walker.WalkDir(func(path string, walkErr error) error { if walkErr != nil { return walkErr @@ -93,16 +97,12 @@ func (c *Core) compileBreakingGraph(ctx context.Context, walker DirWalker) (*bre if err != nil { return err } - physicalPath := filepath.Join(root, filepath.FromSlash(path)) - resolvedPath, err := filepath.EvalSymlinks(physicalPath) - if err == nil { - physicalPath = resolvedPath - } - physicalPath = filepath.Clean(physicalPath) - if physical[physicalPath] { + // An alias has its own FILE identity even when its bytes come from the + // same target. Duplicate declarations remain a compiler error. + if seen[canonicalPath] { return nil } - physical[physicalPath] = true + seen[canonicalPath] = true targets = append(targets, canonicalPath) targetPaths[canonicalPath] = filepath.ToSlash(path) return nil @@ -114,12 +114,12 @@ func (c *Core) compileBreakingGraph(ctx context.Context, walker DirWalker) (*bre if len(targets) == 0 { return newBreakingGraph(), nil } - imports := uniquePhysicalRoots(importRoots) + imports := uniqueLogicalRoots(importRoots) if len(imports) == 0 { imports = []string{root} } compiler := protocompile.Compiler{ - Resolver: wellknownimports.WithStandardImports(&protocompile.SourceResolver{ImportPaths: imports, Accessor: c.openSourceFile}), + Resolver: wellknownimports.WithStandardImports(&protocompile.SourceResolver{ImportPaths: imports, Accessor: c.breakingSourceAccessor(ctx, root)}), SourceInfoMode: protocompile.SourceInfoStandard, } compiled, err := compiler.Compile(ctx, targets...) @@ -129,6 +129,27 @@ func (c *Core) compileBreakingGraph(ctx context.Context, walker DirWalker) (*bre return buildBreakingGraph(compiled, targetPaths), nil } +// Current and baseline walkers own different physical trees. Declared source +// hooks may open imports, while the walker tree remains a bounded local source. +func (c *Core) breakingSourceAccessor(ctx context.Context, root string) func(string) (io.ReadCloser, error) { + return func(filename string) (io.ReadCloser, error) { + if c.importFileAllowed != nil && !c.importFileAllowed(filename) { + return nil, &os.PathError{Op: "open", Path: filename, Err: os.ErrNotExist} + } + if c.sourceFileOpen != nil { + reader, err := c.sourceFileOpen(filename) + if err == nil || !errors.Is(err, os.ErrNotExist) { + return reader, err + } + } + relative, err := filepath.Rel(root, filename) + if err == nil && filepath.IsLocal(relative) { + return sourceview.OpenLocal(ctx, root, relative) + } + return c.openSourceFile(filename) + } +} + func canonicalBreakingPath(root, path string, importRoots []string) (string, error) { physicalPath := filepath.Join(root, filepath.FromSlash(path)) for _, importRoot := range importRoots { @@ -140,14 +161,10 @@ func canonicalBreakingPath(root, path string, importRoots []string) (string, err return filepath.ToSlash(path), nil } -func uniquePhysicalRoots(roots []string) []string { +func uniqueLogicalRoots(roots []string) []string { result := make([]string, 0, len(roots)) seen := make(map[string]bool) for _, root := range roots { - resolved, err := filepath.EvalSymlinks(root) - if err == nil { - root = resolved - } root = filepath.Clean(root) if !seen[root] { seen[root] = true diff --git a/internal/core/core.go b/internal/core/core.go index 10ab85ec..316b3830 100644 --- a/internal/core/core.go +++ b/internal/core/core.go @@ -3,6 +3,7 @@ package core import ( "errors" + "io" "maps" "slices" @@ -24,6 +25,8 @@ type Core struct { inputs Inputs importRoots []string importFileAllowed func(string) bool + sourceFileOpen func(string) (io.ReadCloser, error) + sourceBoundary string fileModules map[string]string managedMode ManagedModeConfig goPackageOutputPrefix string @@ -57,7 +60,9 @@ type Options struct { ImportRoots []string // ImportFileAllowed restricts physical imports and module inputs to the // files selected by dependency metadata. Nil permits every file. - ImportFileAllowed func(string) bool + ImportFileAllowed func(string) bool + // OpenSourceFile opens logical sources within their owning bounded roots. + OpenSourceFile func(string) (io.ReadCloser, error) FileModules map[string]string CurrentProjectGitWalker CurrentProjectGitWalker BreakingCheckConfig BreakingCheckConfig @@ -82,6 +87,7 @@ func New(options Options) *Core { inputs: options.Inputs, importRoots: slices.Clone(options.ImportRoots), importFileAllowed: options.ImportFileAllowed, + sourceFileOpen: options.OpenSourceFile, fileModules: maps.Clone(options.FileModules), currentProjectGitWalker: options.CurrentProjectGitWalker, breakingCheckConfig: options.BreakingCheckConfig, diff --git a/internal/core/generate.go b/internal/core/generate.go index fb976f73..5969d249 100644 --- a/internal/core/generate.go +++ b/internal/core/generate.go @@ -3,6 +3,7 @@ package core import ( "bytes" "context" + "errors" "fmt" stdfs "io/fs" "log/slog" @@ -25,6 +26,7 @@ import ( pluginexecutor "github.com/easyp-tech/easyp/internal/adapters/plugin" "github.com/easyp-tech/easyp/internal/core/path_helpers" "github.com/easyp-tech/easyp/internal/fs/fs" + "github.com/easyp-tech/easyp/internal/sourceview" "github.com/easyp-tech/easyp/internal/version" ) @@ -73,6 +75,7 @@ func (c *Core) PrepareGenerationFiles(ctx context.Context, root string, files [] func (c *Core) prepareGeneration(ctx context.Context, root string, selected []string) (*GenerationPlan, error) { c.logger.Info(ctx, "preparing code generation", slog.String("root", root)) + c.sourceBoundary = root imports := append([]string{}, c.importRoots...) var files []string selection := make(map[string]bool, len(selected)) @@ -90,6 +93,16 @@ func (c *Core) prepareGeneration(ctx context.Context, root string, selected []st } err := fsWalker.WalkDir(func(walkPath string, err error) error { + addedFile := stripPrefix(walkPath, inputFilesDir.Root) + if selected != nil && filepath.Ext(walkPath) == ".proto" && !selection[addedFile] { + return nil + } + if c.importFileAllowed != nil && !c.importFileAllowed(filepath.Join(root, walkPath)) { + return nil + } + if errors.Is(err, sourceview.ErrNestedRepository) && c.sourceFileOpen != nil && c.importFileAllowed != nil && c.importFileAllowed(filepath.Join(root, walkPath)) { + err = nil + } switch { case err != nil: return err @@ -105,7 +118,7 @@ func (c *Core) prepareGeneration(ctx context.Context, root string, selected []st } // Convert to relative path matching proto import format - addedFile := stripPrefix(walkPath, inputFilesDir.Root) + addedFile = stripPrefix(walkPath, inputFilesDir.Root) if selected != nil && !selection[addedFile] { return nil } diff --git a/internal/core/generate_custom_options_test.go b/internal/core/generate_custom_options_test.go new file mode 100644 index 00000000..f2d0406d --- /dev/null +++ b/internal/core/generate_custom_options_test.go @@ -0,0 +1,153 @@ +package core + +import ( + "context" + "fmt" + "io" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/reflect/protodesc" + "google.golang.org/protobuf/types/descriptorpb" + "google.golang.org/protobuf/types/dynamicpb" + "google.golang.org/protobuf/types/pluginpb" + + pluginexecutor "github.com/easyp-tech/easyp/internal/adapters/plugin" + "github.com/easyp-tech/easyp/internal/logger" +) + +func TestGenerateCommandPreservesCustomMethodOptions(t *testing.T) { + t.Parallel() + + for _, tt := range []struct { + name string + managed bool + }{ + {name: "unmanaged"}, + {name: "managed Go package", managed: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeGenerationSources(t, root, map[string]string{ + "mcp/options.proto": `syntax = "proto3"; +package mcp; +import "google/protobuf/descriptor.proto"; +message Method { string name = 1; string title = 2; bool hidden = 4; } +message Service { string namespace = 1; } +extend google.protobuf.MethodOptions { Method method = 91002; } +extend google.protobuf.ServiceOptions { Service service = 91001; } +`, + "api/service.proto": `syntax = "proto3"; +package api.v1; +import "mcp/options.proto"; +option go_package = "example.test/api/v1;apiv1"; +message Request {} +message Response {} +service GeneratorAPI { + option (mcp.service) = {namespace: "catalog"}; + rpc GenerateCode(Request) returns (Response) { option (mcp.method) = {hidden: true}; } + rpc Plugins(Request) returns (Response) { option (mcp.method) = {name: "plugins_list", title: "List plugins"}; } +} +`, + }) + console := &customOptionsConsole{} + executor := pluginexecutor.NewCommandPluginExecutor(console, logger.NewNop()) + app := testCoreWithPlugins([]Plugin{{ + Source: PluginSource{Command: []string{"custom-plugin", "argument with spaces"}}, + Out: "gen", + Options: map[string][]string{"paths": {"source_relative"}}, + }}, executor) + app.pluginWorkDir = root + app.managedMode = ManagedModeConfig{Enabled: tt.managed, Override: []ManagedOverrideRule{{ + FileOption: FileOptionGoPackagePrefix, Value: "example.test/generated", + }}} + + plan, err := app.PrepareGeneration(t.Context(), root) + require.NoError(t, err) + require.NoError(t, plan.Execute(t.Context())) + + require.NotNil(t, console.request) + assert.Equal(t, []string{"api/service.proto"}, console.request.GetFileToGenerate()) + assert.Equal(t, "paths=source_relative", console.request.GetParameter()) + assert.Equal(t, root, console.dir) + assert.Equal(t, []string{"custom-plugin", "argument with spaces"}, console.argv) + assert.Equal(t, []string{"google/protobuf/descriptor.proto", "mcp/options.proto", "api/service.proto"}, descriptorNames(console.request.GetProtoFile())) + generated, err := os.ReadFile(filepath.Join(root, "gen/tools.txt")) + require.NoError(t, err) + assert.Equal(t, "catalog.plugins_list: List plugins\n", string(generated)) + }) + } +} + +// This plugin reads the serialized request with its own extension resolver, as +// an external command does; inspecting the compiler's in-memory options alone +// would miss losses when requests are cloned, managed, or marshaled. +type customOptionsConsole struct { + request *pluginpb.CodeGeneratorRequest + dir string + argv []string +} + +func (*customOptionsConsole) RunCmd(context.Context, string, string, ...string) (string, error) { + return "", fmt.Errorf("unexpected command without stdin") +} + +func (c *customOptionsConsole) RunCmdWithStdin(_ context.Context, dir string, stdin io.Reader, command string, args ...string) (string, error) { + raw, err := io.ReadAll(stdin) + if err != nil { + return "", fmt.Errorf("ReadAll: %w", err) + } + var request pluginpb.CodeGeneratorRequest + if err := proto.Unmarshal(raw, &request); err != nil { + return "", fmt.Errorf("Unmarshal: %w", err) + } + files, err := protodesc.NewFiles(&descriptorpb.FileDescriptorSet{File: request.GetProtoFile()}) + if err != nil { + return "", fmt.Errorf("NewFiles: %w", err) + } + types := dynamicpb.NewTypes(files) + if err := (proto.UnmarshalOptions{Resolver: types}).Unmarshal(raw, &request); err != nil { + return "", fmt.Errorf("Unmarshal: %w", err) + } + c.request, c.dir, c.argv = &request, dir, append([]string{command}, args...) + methodOption, err := types.FindExtensionByName("mcp.method") + if err != nil { + return "", fmt.Errorf("FindExtensionByName: %w", err) + } + serviceOption, err := types.FindExtensionByName("mcp.service") + if err != nil { + return "", fmt.Errorf("FindExtensionByName: %w", err) + } + var content string + for _, file := range request.GetProtoFile() { + if file.GetName() != request.GetFileToGenerate()[0] { + continue + } + for _, service := range file.GetService() { + metadata := proto.GetExtension(service.GetOptions(), serviceOption).(proto.Message).ProtoReflect() + namespace := metadata.Get(metadata.Descriptor().Fields().ByName("namespace")).String() + for _, method := range service.GetMethod() { + metadata := proto.GetExtension(method.GetOptions(), methodOption).(proto.Message).ProtoReflect() + fields := metadata.Descriptor().Fields() + if metadata.Get(fields.ByName("hidden")).Bool() { + continue + } + name := metadata.Get(fields.ByName("name")).String() + title := metadata.Get(fields.ByName("title")).String() + content += namespace + "." + name + ": " + title + "\n" + } + } + } + response, err := proto.Marshal(&pluginpb.CodeGeneratorResponse{File: []*pluginpb.CodeGeneratorResponse_File{{ + Name: proto.String("tools.txt"), Content: proto.String(content), + }}}) + if err != nil { + return "", fmt.Errorf("Marshal: %w", err) + } + return string(response), nil +} diff --git a/internal/core/path_helpers/v1_source.go b/internal/core/path_helpers/v1_source.go index c1888f6f..b21b51f2 100644 --- a/internal/core/path_helpers/v1_source.go +++ b/internal/core/path_helpers/v1_source.go @@ -16,16 +16,28 @@ func ShouldSkipV1SourceDir(root, path string) bool { if err != nil || !info.IsDir() { return false } - if strings.HasPrefix(filepath.Base(path), ".") { + if HiddenOrVendorSourcePath(root, path) { return true } - if filepath.Base(path) == "easyp_vendor" { - return true - } - for _, name := range []string{"protobuf.mod", "buf.work.yaml", "buf.yaml"} { + for _, name := range []string{".git", "protobuf.mod", "buf.work.yaml", "buf.yaml"} { if _, err := os.Stat(filepath.Join(path, name)); err == nil { return true } } return false } + +// HiddenOrVendorSourcePath reports lexical hidden and vendored descendants. +// It does not inspect metadata ownership or resolve aliases. +func HiddenOrVendorSourcePath(root, path string) bool { + relative, err := filepath.Rel(root, path) + if err != nil || !filepath.IsLocal(relative) { + return false + } + for _, part := range strings.Split(relative, string(filepath.Separator)) { + if part != "." && (strings.HasPrefix(part, ".") || part == "easyp_vendor") { + return true + } + } + return false +} diff --git a/internal/core/proto_info_read.go b/internal/core/proto_info_read.go index 8d38f948..1071bcd9 100644 --- a/internal/core/proto_info_read.go +++ b/internal/core/proto_info_read.go @@ -15,7 +15,7 @@ import ( ) func (c *Core) protoInfoRead(ctx context.Context, fs FS, path string) (ProtoInfo, error) { - f, err := fs.Open(path) + f, err := c.openDiskSource(fs, path) if err != nil { return ProtoInfo{}, fmt.Errorf("Open: %w", err) } @@ -98,7 +98,7 @@ func (c *Core) openImportFile(disk FS, importName string) (io.ReadCloser, error) allowed = c.importFileAllowed(filepath.Join(rooted.RootPath(), importName)) } if allowed { - f, err = disk.Open(importName) + f, err = c.openDiskSource(disk, importName) } else { err = os.ErrNotExist } @@ -152,3 +152,17 @@ func (c *Core) sourceImportPath(disk FS, name string) string { } return filepath.ToSlash(best) } + +func (c *Core) openDiskSource(disk FS, name string) (io.ReadCloser, error) { + if rooted, ok := disk.(interface{ RootPath() string }); ok && c.sourceFileOpen != nil { + full := filepath.Join(rooted.RootPath(), filepath.FromSlash(name)) + if c.importFileAllowed != nil && !c.importFileAllowed(full) { + return nil, &os.PathError{Op: "open", Path: name, Err: os.ErrNotExist} + } + file, err := c.sourceFileOpen(full) + if err == nil || !errors.Is(err, os.ErrNotExist) { + return file, err + } + } + return disk.Open(name) +} diff --git a/internal/core/source_accessor.go b/internal/core/source_accessor.go index 12e04f79..a563d070 100644 --- a/internal/core/source_accessor.go +++ b/internal/core/source_accessor.go @@ -1,8 +1,12 @@ package core import ( + "context" "io" "os" + "path/filepath" + + "github.com/easyp-tech/easyp/internal/sourceview" ) // openSourceFile is shared by descriptor compilation and syntax-based import lookup. @@ -10,5 +14,18 @@ func (c *Core) openSourceFile(path string) (io.ReadCloser, error) { if c.importFileAllowed != nil && !c.importFileAllowed(path) { return nil, &os.PathError{Op: "open", Path: path, Err: os.ErrNotExist} } - return os.Open(path) + if c.sourceFileOpen != nil { + return c.sourceFileOpen(path) + } + roots := append([]string{c.sourceBoundary}, c.importRoots...) + for _, root := range roots { + if root == "" { + continue + } + relative, err := filepath.Rel(root, path) + if err == nil && filepath.IsLocal(relative) { + return sourceview.OpenLocal(context.Background(), root, relative) + } + } + return sourceview.OpenLocal(context.Background(), filepath.Dir(path), filepath.Base(path)) } diff --git a/internal/fs/fs/adapter.go b/internal/fs/fs/adapter.go index e9d75c7b..26df8d0e 100644 --- a/internal/fs/fs/adapter.go +++ b/internal/fs/fs/adapter.go @@ -1,10 +1,13 @@ package fs import ( + "context" "io" "io/fs" "os" "path/filepath" + + "github.com/easyp-tech/easyp/internal/sourceview" ) type FSAdapter struct { @@ -14,7 +17,7 @@ type FSAdapter struct { } func (a *FSAdapter) Open(name string) (io.ReadCloser, error) { - return a.FS.Open(name) + return sourceview.OpenLocal(context.Background(), a.rootDir, name) } func (a *FSAdapter) Create(name string) (io.WriteCloser, error) { diff --git a/internal/fs/fs/dir_walker.go b/internal/fs/fs/dir_walker.go index 17886945..29fc3a37 100644 --- a/internal/fs/fs/dir_walker.go +++ b/internal/fs/fs/dir_walker.go @@ -1,10 +1,12 @@ package fs import ( + "context" "io" - "io/fs" "os" "path/filepath" + + "github.com/easyp-tech/easyp/internal/sourceview" ) type FS interface { @@ -36,9 +38,10 @@ type FSWalker struct { } func (w *FSWalker) WalkDir(callback func(path string, err error) error) error { - err := fs.WalkDir(w.FS, w.path, func(path string, d fs.DirEntry, err error) error { + return sourceview.WalkLocal(context.Background(), w.rootDir, w.path, func(path string, resolved sourceview.Resolution, err error) error { + if err != nil && filepath.Ext(path) != ".proto" && path != w.path { + return nil + } return callback(path, err) }) - - return err } diff --git a/internal/generation/aliases_test.go b/internal/generation/aliases_test.go new file mode 100644 index 00000000..eb42c7c5 --- /dev/null +++ b/internal/generation/aliases_test.go @@ -0,0 +1,113 @@ +package generation + +import ( + "os" + "path/filepath" + "testing" + + "github.com/easyp-tech/easyp/internal/logger" + "github.com/stretchr/testify/require" + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/types/descriptorpb" +) + +func TestAutomaticSelectionAcceptsInternalLinkedGenerator(t *testing.T) { + t.Parallel() + root := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(root, "generator-source.yaml"), []byte("version: v1\n"), 0o644)) + require.NoError(t, os.Symlink("generator-source.yaml", filepath.Join(root, "easyp.gen.yaml"))) + configs, err := selectGenerateConfigs(Request{WorkDir: root, WorkspaceRoot: root}) + require.NoError(t, err) + require.Equal(t, []string{filepath.Join(root, "easyp.gen.yaml")}, configs) +} + +func TestModuleDiscoveryRejectsDuplicateDirectoryAliasIdentity(t *testing.T) { + t.Parallel() + root := t.TempDir() + require.NoError(t, os.Mkdir(filepath.Join(root, "real"), 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(root, "real/protobuf.mod"), []byte("module example.com/api\nroots .\n"), 0o644)) + require.NoError(t, os.Symlink("real", filepath.Join(root, "alias"))) + _, err := findV1LocalModuleByName(root, "example.com/api") + require.ErrorContains(t, err, "declared in both") +} + +func TestGenerateDirectoryAndMetadataAliasesPreserveDescriptorNames(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "sources/api/model.proto", "syntax = \"proto3\"; package model.v1; message Model {}") + writeV1GenerateFixture(t, root, "generator-source.yaml", "version: v1\n") + writeV1GenerateFixture(t, root, "manifest-source", "module example.com/api\nroots proto\n") + require.NoError(t, os.Symlink("sources", filepath.Join(root, "proto"))) + require.NoError(t, os.Symlink("generator-source.yaml", filepath.Join(root, "easyp.gen.yaml"))) + require.NoError(t, os.Symlink("manifest-source", filepath.Join(root, "protobuf.mod"))) + require.NoError(t, os.Symlink("missing", filepath.Join(root, "sources/unused.txt"))) + output := filepath.Join(root, "descriptor.pb") + require.NoError(t, Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root, WorkspaceRoot: root, DescriptorSetOut: output})) + raw, err := os.ReadFile(output) + require.NoError(t, err) + var descriptors descriptorpb.FileDescriptorSet + require.NoError(t, proto.Unmarshal(raw, &descriptors)) + require.Len(t, descriptors.File, 1) + require.Equal(t, "api/model.proto", descriptors.File[0].GetName()) +} + +func TestGenerateSelectedFilesSkipsDanglingUnselectedAlias(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "api/model.proto", "syntax = \"proto3\"; package model.v1; message Model {}") + writeV1GenerateFixture(t, root, "easyp.gen.yaml", "version: v1\ngenerate:\n paths: [api]\n") + require.NoError(t, os.Symlink("missing", filepath.Join(root, "unselected.proto"))) + require.NoError(t, Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root, WorkspaceRoot: root, DescriptorSetOut: filepath.Join(root, "descriptor.pb")})) +} + +func TestGenerateDeclaredDependencyAliasAcrossNestedRepository(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "protobuf.mod", "module example.test/app\nroots .\nrequire example.test/dep\nreplace example.test/dep => ./dep\n") + writeV1GenerateFixture(t, root, "dep/.git", "gitdir: /unused\n") + writeV1GenerateFixture(t, root, "dep/buf.yaml", "version: v2\nmodules:\n - path: .\n includes: [selected]\n") + writeV1GenerateFixture(t, root, "dep/selected/model.proto", "syntax = \"proto3\"; package model.v1; message Model {}\n") + writeV1GenerateFixture(t, root, "easyp.gen.yaml", "version: v1\ngenerate:\n paths: [alias.proto]\n packages: [model.v1]\n") + require.NoError(t, os.Symlink("dep/selected/model.proto", filepath.Join(root, "alias.proto"))) + require.NoError(t, Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root, WorkspaceRoot: root, DescriptorSetOut: filepath.Join(root, "descriptor.pb")})) +} + +func TestModuleDiscoveryIgnoresUnselectedProtoAliasFailures(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "protobuf.mod", "module example.test/app\nroots api\n") + writeV1GenerateFixture(t, root, "dep/protobuf.mod", "module example.test/dep\nroots allowed\n") + require.NoError(t, os.MkdirAll(filepath.Join(root, "dep/excluded"), 0o755)) + require.NoError(t, os.Symlink("../../../outside.proto", filepath.Join(root, "dep/excluded/alias.proto"))) + found, err := findV1LocalModuleByName(root, "example.test/dep") + require.NoError(t, err) + require.Equal(t, filepath.Join(root, "dep"), found) +} + +func TestGenerateSelectedBufDependencyIgnoresExcludedAliasFailures(t *testing.T) { + t.Parallel() + for _, tt := range []struct{ name, metadata string }{ + {name: "v1 excluded", metadata: "version: v1\nbuild:\n excludes: [excluded]\n"}, + {name: "v2 not included", metadata: "version: v2\nmodules:\n - path: .\n includes: [allowed]\n"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "protobuf.mod", "module example.test/app\nroots api\nrequire example.test/dep\nreplace example.test/dep => ./dep\n") + writeV1GenerateFixture(t, root, "api/client.proto", "syntax = \"proto3\"; package app; message Client {}\n") + writeV1GenerateFixture(t, root, "dep/buf.yaml", tt.metadata) + writeV1GenerateFixture(t, root, "dep/allowed/model.proto", "syntax = \"proto3\"; package model; message Model {}\n") + writeV1GenerateFixture(t, root, "easyp.gen.yaml", "version: v1\ngenerate:\n modules: [example.test/dep]\n") + require.NoError(t, os.MkdirAll(filepath.Join(root, "dep/excluded"), 0o755)) + require.NoError(t, os.Symlink("../../../outside.proto", filepath.Join(root, "dep/excluded/alias.proto"))) + output := filepath.Join(root, "descriptor.pb") + require.NoError(t, Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root, WorkspaceRoot: root, DescriptorSetOut: output})) + raw, err := os.ReadFile(output) + require.NoError(t, err) + var descriptors descriptorpb.FileDescriptorSet + require.NoError(t, proto.Unmarshal(raw, &descriptors)) + require.Len(t, descriptors.File, 1) + require.Equal(t, "allowed/model.proto", descriptors.File[0].GetName()) + }) + } +} diff --git a/internal/generation/descriptor_set.go b/internal/generation/descriptor_set.go index f28d4e03..9893414c 100644 --- a/internal/generation/descriptor_set.go +++ b/internal/generation/descriptor_set.go @@ -56,13 +56,11 @@ func generateV1DescriptorSet(ctx context.Context, log logger.Logger, cache modul func prepareDescriptorTargets(ctx context.Context, log logger.Logger, cache modules.Cache, request Request, targets []generationTarget) ([]preparedDescriptorTarget, error) { prepared := make([]preparedDescriptorTarget, 0, len(targets)) - seen := make(map[string]bool) - requested := make(map[string][]string) - matched := make(map[string]map[string]bool) + seen := make(map[string]v1ModuleSelection) + requested := make(map[string]*sourceSelectorMatches) for _, target := range targets { - if len(target.config.Generate.Packages) > 0 { - requested[target.configPath] = target.config.Generate.Packages - matched[target.configPath] = make(map[string]bool) + if target.config.Generate.HasSourceSelectors() { + requested[target.configPath] = newSourceSelectorMatches(target.config.Generate.Packages, target.config.Generate.Paths) } } for _, target := range targets { @@ -73,24 +71,29 @@ func prepareDescriptorTargets(ctx context.Context, log logger.Logger, cache modu if err != nil { return nil, fmt.Errorf("resolveV1GenerationModule for %s: %w", target.configPath, err) } - // Options-only selections still validate their graph in frozen mode, but - // have no generation work unless descriptor export was requested. - if len(target.config.Plugins) == 0 && len(target.config.Generate.Packages) == 0 && request.DescriptorSetOut == "" && request.DescriptorSetOutDir == "" { + key := filepath.Clean(target.configPath) + "\x00" + filepath.Clean(selected.directory) + if previous, exists := seen[key]; exists { + if !sameSourceSelectors(previous.packages, target.module.packages) || !sameSourceSelectors(previous.paths, target.module.paths) { + return nil, fmt.Errorf("%s: generate.modules[%d] and generate.modules[%d] select module %q with conflicting filters", target.configPath, previous.index, target.module.index, selected.module.Name) + } continue } - key := filepath.Clean(target.configPath) + "\x00" + filepath.Clean(selected.directory) - if seen[key] { + seen[key] = target.module + moduleSelectors := newSourceSelectorMatches(target.module.packages, target.module.paths) + // Options-only selections still validate their graph in frozen mode, but + // have no generation work unless descriptor export was requested. + if len(target.config.Plugins) == 0 && !target.config.Generate.HasSourceSelectors() && request.DescriptorSetOut == "" && request.DescriptorSetOutDir == "" { continue } - seen[key] = true var files []string - if len(target.config.Generate.Packages) > 0 { - var matches map[string]bool - files, matches, err = selectedPackageFiles(ctx, selected, target.config.Generate.Packages) + if selectors, filtered := requested[target.configPath]; filtered { + files, err = selectedSourceFiles(ctx, selected, selectors, moduleSelectors) if err != nil { - return nil, fmt.Errorf("package selection in %s: %w", target.configPath, err) + return nil, fmt.Errorf("selectedSourceFiles for %s: %w", target.configPath, err) + } + if err := moduleSelectors.validateMatches(fmt.Sprintf("generate.modules[%d]", target.module.index)); err != nil { + return nil, fmt.Errorf("%s: %w", target.configPath, err) } - maps.Copy(matched[target.configPath], matches) if len(files) == 0 { continue } @@ -126,19 +129,31 @@ func prepareDescriptorTargets(ctx context.Context, log logger.Logger, cache modu prepared = append(prepared, preparedDescriptorTarget{target: target, selected: selected, plan: plan, full: full, owners: owners, versions: versions, label: label}) } for _, path := range slices.Sorted(maps.Keys(requested)) { - var unknown []string - for _, name := range requested[path] { - if !matched[path][name] && !slices.Contains(unknown, name) { - unknown = append(unknown, name) - } - } - if len(unknown) > 0 { - return nil, fmt.Errorf("%s: generate.packages did not match any selected module source files: %s", path, strings.Join(unknown, ", ")) + selectors := requested[path] + if err := selectors.validateMatches("generate"); err != nil { + return nil, fmt.Errorf("%s: %w", path, err) } } return prepared, nil } +func sameSourceSelectors(first, second []string) bool { + a, b := slices.Clone(first), slices.Clone(second) + slices.Sort(a) + slices.Sort(b) + return slices.Equal(slices.Compact(a), slices.Compact(b)) +} + +func unmatchedSourceSelectors(requested []string, matched map[string]bool) []string { + var unknown []string + for _, name := range requested { + if !matched[name] && !slices.Contains(unknown, name) { + unknown = append(unknown, name) + } + } + return unknown +} + func combineDescriptorTargets(targets []preparedDescriptorTarget, includeImports bool) (*descriptorpb.FileDescriptorSet, error) { combined := &descriptorpb.FileDescriptorSet{} files := make(map[string]*descriptorpb.FileDescriptorProto) diff --git a/internal/generation/discovery.go b/internal/generation/discovery.go index 539e8dd5..8fe159e6 100644 --- a/internal/generation/discovery.go +++ b/internal/generation/discovery.go @@ -2,7 +2,6 @@ package generation import ( "fmt" - "os" "path/filepath" "slices" @@ -49,7 +48,7 @@ func selectGenerateConfigs(request Request) ([]string, error) { return nil, lookupErr } if legacy != "" { - raw, readErr := os.ReadFile(legacy) + raw, readErr := workspace.ReadFile(request.WorkspaceRoot, legacy) if readErr != nil { return nil, readErr } @@ -59,12 +58,5 @@ func selectGenerateConfigs(request Request) ([]string, error) { } return nil, fmt.Errorf("no selected easyp.gen.yaml; choose --project (repeatable), or --all for recursive generation") } - info, err := os.Lstat(path) - if err != nil { - return nil, err - } - if info.Mode()&os.ModeSymlink != 0 { - return nil, fmt.Errorf("automatic generation selection does not follow symlink config %s; select its project explicitly", path) - } return []string{filepath.Clean(path)}, nil } diff --git a/internal/generation/generate.go b/internal/generation/generate.go index b3ea9e9e..5e58885a 100644 --- a/internal/generation/generate.go +++ b/internal/generation/generate.go @@ -12,6 +12,7 @@ import ( "github.com/easyp-tech/easyp/internal/core/path_helpers" "github.com/easyp-tech/easyp/internal/logger" "github.com/easyp-tech/easyp/internal/modules" + "github.com/easyp-tech/easyp/internal/sourceview" "github.com/easyp-tech/easyp/internal/workspace" ) @@ -84,16 +85,13 @@ func Run(ctx context.Context, log logger.Logger, cache modules.Cache, request Re if err := inheritV1GenerateOptions(request.WorkspaceRoot, configPath, &gen); err != nil { return fmt.Errorf("inheritV1GenerateOptions: %w", err) } - if len(gen.Plugins) == 0 && len(gen.Generate.Packages) == 0 && !exportDescriptors && !request.Frozen { + if len(gen.Plugins) == 0 && !gen.Generate.HasSourceSelectors() && !exportDescriptors && !request.Frozen { continue } modules, err := selectV1Modules(request.WorkspaceRoot, filepath.Dir(configPath), gen.Generate.Modules) if err != nil { return fmt.Errorf("%s: %w", configPath, err) } - if err := v1.ValidatePackageSelectors(gen.Generate.Packages); err != nil { - return fmt.Errorf("%s: %w", configPath, err) - } for _, module := range modules { descriptorTargets = append(descriptorTargets, generationTarget{configPath: configPath, config: gen, module: module}) } @@ -101,7 +99,8 @@ func Run(ctx context.Context, log logger.Logger, cache modules.Cache, request Re if request.AllProjects { selected := descriptorTargets[:0] for _, target := range descriptorTargets { - if len(target.config.Plugins) == 0 && len(target.config.Generate.Modules) == 0 { + if len(target.config.Plugins) == 0 && len(target.config.Generate.Modules) == 0 && + !target.config.Generate.HasSourceSelectors() { inherited, err := isOptionsOnlyParent(target.configPath, target.config, configs) if err != nil { return fmt.Errorf("isOptionsOnlyParent: %w", err) @@ -217,7 +216,7 @@ func isOptionsOnlyParent(configPath string, gen v1.Generate, configs []string) ( } hasOwnProto := false - err := filepath.WalkDir(dir, func(path string, entry fs.DirEntry, walkErr error) error { + err := workspace.Walk(dir, func(path string, entry fs.DirEntry, walkErr error) error { if walkErr != nil { return walkErr } @@ -242,14 +241,26 @@ func discoverV1GenerateConfigs(root, project string) ([]string, error) { project = filepath.Join(root, project) } path := filepath.Join(project, v1.GenerateFile) - if _, err := os.Stat(path); err != nil { + boundary := root + relative, err := filepath.Rel(boundary, path) + if err != nil { + return nil, fmt.Errorf("Rel: %w", err) + } + if !filepath.IsLocal(relative) { + boundary = project + relative = v1.GenerateFile + } + if _, err := sourceview.ResolveLocal(context.Background(), boundary, relative); err != nil { return nil, fmt.Errorf("find project generator %s: %w", path, err) } return []string{path}, nil } var paths []string - err := filepath.WalkDir(root, func(path string, entry fs.DirEntry, walkErr error) error { + err := workspace.Walk(root, func(path string, entry fs.DirEntry, walkErr error) error { if walkErr != nil { + if filepath.Base(path) != v1.GenerateFile { + return nil + } return walkErr } if entry.IsDir() { @@ -267,7 +278,7 @@ func discoverV1GenerateConfigs(root, project string) ([]string, error) { } func readV1GenerateConfig(path string) (v1.Generate, error) { - raw, err := os.ReadFile(path) + raw, err := workspace.ReadFileAt(path) if err != nil { return v1.Generate{}, fmt.Errorf("ReadFile: %w", err) } diff --git a/internal/generation/inherit.go b/internal/generation/inherit.go index 84bb950e..43a363bd 100644 --- a/internal/generation/inherit.go +++ b/internal/generation/inherit.go @@ -8,6 +8,7 @@ import ( "path/filepath" v1 "github.com/easyp-tech/easyp/internal/config/v1" + "github.com/easyp-tech/easyp/internal/workspace" ) // inheritV1GenerateOptions reads only options from ancestor generator files. @@ -46,7 +47,7 @@ func inheritV1GenerateOptions(repoRoot, configPath string, gen *v1.Generate) err } func readOptionalFile(path string) ([]byte, bool, error) { - raw, err := os.ReadFile(path) + raw, err := workspace.ReadFileAt(path) if errors.Is(err, os.ErrNotExist) { return nil, false, nil } diff --git a/internal/generation/module.go b/internal/generation/module.go index 54bb820e..2c554562 100644 --- a/internal/generation/module.go +++ b/internal/generation/module.go @@ -38,6 +38,7 @@ func prepareV1ModuleCore(log logger.Logger, request Request, configPath, moduleD options.PluginWorkDir = request.WorkDir options.ImportRoots = importRoots.Paths() options.ImportFileAllowed = allSources.FileAllowed() + options.OpenSourceFile = allSources.OpenSourceFile if options.ManagedModeConfig.Enabled || options.ManagedModeConfig.GoPackageOnly { roots := append(modules.SourceRoots(nil), importRoots...) roots = append(roots, sources...) diff --git a/internal/generation/module_filters_test.go b/internal/generation/module_filters_test.go new file mode 100644 index 00000000..c65825ab --- /dev/null +++ b/internal/generation/module_filters_test.go @@ -0,0 +1,178 @@ +package generation + +import ( + "fmt" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/easyp-tech/easyp/internal/logger" +) + +func TestScopedSelectorsValidateWithoutPlugins(t *testing.T) { + t.Parallel() + for _, tt := range []struct{ name, field string }{ + {name: "paths", field: "paths"}, + {name: "packages", field: "packages"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "protobuf.mod", "module example.com/app\n") + writeV1GenerateFixture(t, root, "easyp.gen.yaml", "generate:\n modules:\n - module: .\n "+tt.field+": [missing]\n") + writeV1GenerateFixture(t, root, "item.proto", "syntax = \"proto3\"; package item.v1; message Item {}") + + err := Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root}) + + require.ErrorContains(t, err, "generate.modules[0]."+tt.field) + }) + } +} + +func TestAllChecksScopedParentSelectorsBeforeChildPlugins(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "protobuf.mod", "module example.com/parent\n") + writeV1GenerateFixture(t, root, "easyp.gen.yaml", "generate:\n modules: [{module: ., paths: [missing]}]\n") + writeV1GenerateFixture(t, root, "child/protobuf.mod", "module example.com/child\n") + writeV1GenerateFixture(t, root, "child/easyp.gen.yaml", "plugins: [{command: [sh, -c, 'touch plugin-ran'], out: gen}]\n") + writeV1GenerateFixture(t, root, "child/item.proto", "syntax = \"proto3\"; package child.v1; message Item {}") + + err := Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root, AllProjects: true}) + + require.ErrorContains(t, err, "generate.modules[0].paths") + assert.NoFileExists(t, filepath.Join(root, "child/plugin-ran")) +} + +func TestModuleDuplicateFiltersAreUnambiguous(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name, first, second string + conflict bool + }{ + {name: "same_unfiltered_forms", first: "contracts", second: "{module: contracts, paths: [], packages: []}"}, + {name: "reordered_repeated_filters", first: "{module: contracts, paths: [proto, proto/api]}", second: "{module: example.com/contracts, paths: [proto/api, proto, proto]}"}, + {name: "unfiltered_then_filtered", first: "contracts", second: "{module: contracts, paths: [proto/api/first.proto]}", conflict: true}, + {name: "filtered_then_unfiltered", first: "{module: contracts, paths: [proto/api/first.proto]}", second: "contracts", conflict: true}, + {name: "directory_then_identity", first: "{module: contracts, paths: [proto/api/first.proto]}", second: "{module: example.com/contracts, paths: [proto/api/second.proto]}", conflict: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "easyp.gen.yaml", fmt.Sprintf("generate:\n modules: [%s, %s]\n", tt.first, tt.second)) + writeV1GenerateFixture(t, root, "contracts/protobuf.mod", "module example.com/contracts\nroots proto\n") + writeV1GenerateFixture(t, root, "contracts/proto/api/first.proto", "syntax = \"proto3\"; package api.v1; message First {}") + writeV1GenerateFixture(t, root, "contracts/proto/api/second.proto", "syntax = \"proto3\"; package api.v1; message Second {}") + out := filepath.Join(root, "selected.pb") + previous := []byte("previous descriptor") + require.NoError(t, os.WriteFile(out, previous, 0o600)) + + err := Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root, DescriptorSetOut: out}) + + if tt.conflict { + require.ErrorContains(t, err, "generate.modules[0] and generate.modules[1]") + require.ErrorContains(t, err, "conflicting filters") + actual, readErr := os.ReadFile(out) + require.NoError(t, readErr) + assert.Equal(t, previous, actual) + return + } + require.NoError(t, err) + assert.ElementsMatch(t, []string{"api/first.proto", "api/second.proto"}, descriptorNames(readDescriptorSet(t, out))) + }) + } +} + +func TestModulePathsUseLogicalRootAlias(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name, path string + reject bool + }{ + {name: "logical_alias", path: "proto/api"}, + {name: "physical_target_is_not_a_selector", path: "source/api", reject: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "protobuf.mod", "module example.com/app\nroots proto\n") + writeV1GenerateFixture(t, root, "easyp.gen.yaml", "generate:\n modules: [{module: ., paths: ["+tt.path+"]}]\n") + writeV1GenerateFixture(t, root, "source/api/model.proto", "syntax = \"proto3\"; package api.v1; message Model {}") + require.NoError(t, os.Symlink("source", filepath.Join(root, "proto"))) + out := filepath.Join(root, "selected.pb") + + err := Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root, DescriptorSetOut: out}) + + if tt.reject { + require.ErrorContains(t, err, "generate.modules[0].paths") + assert.NoFileExists(t, out) + return + } + require.NoError(t, err) + assert.Equal(t, []string{"api/model.proto"}, descriptorNames(readDescriptorSet(t, out))) + }) + } +} + +func TestModuleFiltersIntersectGlobalSelectors(t *testing.T) { + t.Parallel() + root := t.TempDir() + files := map[string]string{ + "easyp.gen.yaml": "version: v1\ngenerate:\n modules:\n - module: first\n paths: [proto/api/first.proto]\n packages: [first.v1]\n - module: second\n paths: [proto/api/second.proto]\n packages: [second.v1]\n paths: [proto/api]\n packages: [first.v1, second.v1]\n", + "first/protobuf.mod": "module example.com/first\nroots proto\n", + "second/protobuf.mod": "module example.com/second\nroots proto\n", + "first/proto/api/first.proto": "syntax = \"proto3\"; package first.v1; import \"common/type.proto\"; message First { common.v1.Type value = 1; }", + "second/proto/api/second.proto": "syntax = \"proto3\"; package second.v1; import \"common/type.proto\"; message Second { common.v1.Type value = 1; }", + "first/proto/api/ignored.proto": "syntax = \"proto3\"; package first.v1; message Broken {", + "second/proto/api/ignored.proto": "syntax = \"proto3\"; package second.v1; message Broken {", + } + for _, module := range []string{"first", "second"} { + files[module+"/proto/common/type.proto"] = "syntax = \"proto3\"; package common.v1; message Type {}" + } + for name, content := range files { + writeV1GenerateFixture(t, root, name, content) + } + out := filepath.Join(root, "selected.pb") + + require.NoError(t, Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root, DescriptorSetOut: out, IncludeImports: true})) + + assert.ElementsMatch(t, []string{"api/first.proto", "api/second.proto", "common/type.proto"}, descriptorNames(readDescriptorSet(t, out))) +} + +func TestModuleFilterCannotMatchAnotherModule(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "easyp.gen.yaml", "generate:\n modules:\n - module: first\n paths: [proto/api/second.proto]\n - second\nplugins: [{command: [sh, -c, 'touch plugin-ran'], out: gen}]\n") + for _, module := range []string{"first", "second"} { + writeV1GenerateFixture(t, root, module+"/protobuf.mod", "module example.com/"+module+"\nroots proto\n") + writeV1GenerateFixture(t, root, module+"/proto/api/"+module+".proto", "syntax = \"proto3\"; package "+module+".v1; message Item {}") + } + out := filepath.Join(root, "selected.pb") + previous := []byte("previous descriptor") + require.NoError(t, os.WriteFile(out, previous, 0o600)) + + err := Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root, DescriptorSetOut: out}) + + require.ErrorContains(t, err, "generate.modules[0].paths") + assert.NoFileExists(t, filepath.Join(root, "plugin-ran")) + actual, readErr := os.ReadFile(out) + require.NoError(t, readErr) + assert.Equal(t, previous, actual) +} + +func TestGlobalPathsUseModuleDirectoryWithoutExplicitModule(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "protobuf.mod", "module example.com/service\nroots api\n") + writeV1GenerateFixture(t, root, "easyp.gen.yaml", "generate:\n paths: [api/easyp]\n") + writeV1GenerateFixture(t, root, "api/easyp/generator/v1/generator.proto", "syntax = \"proto3\"; package easyp.generator.v1; message Generator {}") + writeV1GenerateFixture(t, root, "api/other/bad.proto", "package invalid.v1; message {") + out := filepath.Join(root, "selected.pb") + + require.NoError(t, Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root, DescriptorSetOut: out})) + + assert.Equal(t, []string{"easyp/generator/v1/generator.proto"}, descriptorNames(readDescriptorSet(t, out))) +} diff --git a/internal/generation/packages.go b/internal/generation/packages.go index 42b69af9..a7150071 100644 --- a/internal/generation/packages.go +++ b/internal/generation/packages.go @@ -1,52 +1,102 @@ package generation import ( - "bytes" "context" "fmt" - "os" "path/filepath" "slices" "strings" - "text/scanner" + v1 "github.com/easyp-tech/easyp/internal/config/v1" "github.com/easyp-tech/easyp/internal/modules" + "github.com/easyp-tech/easyp/internal/protosource" ) -// selectedPackageFiles reads package declarations without compiling unrelated -// files. Malformed selected files and required imports are rejected by the real -// compiler later; an unrelated broken declaration is not a generation target. -func selectedPackageFiles(ctx context.Context, selected v1GenerationModule, packages []string) ([]string, map[string]bool, error) { +type sourceSelectorMatches struct { + packages []string + paths []string + packageMatches map[string]bool + pathMatches map[string]bool +} + +func newSourceSelectorMatches(packages, paths []string) *sourceSelectorMatches { + return &sourceSelectorMatches{packages: packages, paths: paths, packageMatches: make(map[string]bool), pathMatches: make(map[string]bool)} +} + +func (s *sourceSelectorMatches) matchesPath(name string) bool { + return len(s.paths) == 0 || slices.ContainsFunc(s.paths, func(selector string) bool { return v1.PathSelectorMatches(selector, name) }) +} + +func (s *sourceSelectorMatches) matchesPackage(name string) bool { + return len(s.packages) == 0 || slices.Contains(s.packages, name) +} + +func (s *sourceSelectorMatches) recordMatches(packageName, path string) { + if len(s.packages) > 0 { + s.packageMatches[packageName] = true + } + for _, selector := range s.paths { + if v1.PathSelectorMatches(selector, path) { + s.pathMatches[selector] = true + } + } +} + +func (s *sourceSelectorMatches) validateMatches(section string) error { + if unknown := unmatchedSourceSelectors(s.packages, s.packageMatches); len(unknown) > 0 { + return fmt.Errorf("%s.packages did not match any selected module source files: %s", section, strings.Join(unknown, ", ")) + } + if unknown := unmatchedSourceSelectors(s.paths, s.pathMatches); len(unknown) > 0 { + return fmt.Errorf("%s.paths did not match any selected module source files: %s", section, strings.Join(unknown, ", ")) + } + return nil +} + +// selectedSourceFiles intersects literal module-relative paths and exact package +// names without compiling unrelated files. Package declarations are read only +// when needed and after the path filter; required imports are compiled later. +func selectedSourceFiles(ctx context.Context, selected v1GenerationModule, project, module *sourceSelectorMatches) ([]string, error) { roots, err := modules.ModuleSources(selected.directory, selected.module) if err != nil { - return nil, nil, fmt.Errorf("ModuleSources: %w", err) + return nil, fmt.Errorf("ModuleSources: %w", err) } - requested := make(map[string]bool, len(packages)) - for _, name := range packages { - requested[name] = true - } - matched := make(map[string]bool) + allRoots := append(append(modules.SourceRoots(nil), roots...), selected.dependencies...) seen := make(map[string]bool) var files []string for _, root := range roots { - err := root.Walk(func(path string) error { + err := allRoots.WalkSelected(root, func(path string) bool { + relative, err := filepath.Rel(selected.directory, path) + return err == nil && project.matchesPath(filepath.ToSlash(relative)) && module.matchesPath(filepath.ToSlash(relative)) + }, func(path string) error { if err := ctx.Err(); err != nil { return fmt.Errorf("Err: %w", err) } - raw, err := os.ReadFile(path) + modulePath, err := filepath.Rel(selected.directory, path) if err != nil { - return fmt.Errorf("ReadFile: %w", err) + return fmt.Errorf("Rel: %w", err) } - name := sourcePackage(raw) - if !requested[name] { + modulePath = filepath.ToSlash(modulePath) + if !project.matchesPath(modulePath) || !module.matchesPath(modulePath) { return nil } + packageName := "" + if len(project.packages) > 0 || len(module.packages) > 0 { + raw, err := allRoots.ReadSourceFile(path) + if err != nil { + return fmt.Errorf("ReadFile: %w", err) + } + packageName = protosource.Package(raw) + if !project.matchesPackage(packageName) || !module.matchesPackage(packageName) { + return nil + } + } + project.recordMatches(packageName, modulePath) + module.recordMatches(packageName, modulePath) relative, err := filepath.Rel(root.Path, path) if err != nil { return fmt.Errorf("Rel: %w", err) } relative = filepath.ToSlash(relative) - matched[name] = true if !seen[relative] { seen[relative] = true files = append(files, relative) @@ -54,59 +104,9 @@ func selectedPackageFiles(ctx context.Context, selected v1GenerationModule, pack return nil }) if err != nil { - return nil, nil, fmt.Errorf("WalkProtoFiles: %w", err) + return nil, fmt.Errorf("Walk: %w", err) } } slices.Sort(files) - return files, matched, nil -} - -// sourcePackage lexes only top-level package declarations. Scanner tokens keep -// strings and comments opaque, so an option containing "package" cannot select -// a file. It intentionally does not validate unrelated message definitions. -func sourcePackage(raw []byte) string { - var lex scanner.Scanner - lex.Init(bytes.NewReader(raw)) - lex.Mode = scanner.ScanIdents | scanner.ScanStrings | scanner.ScanChars | scanner.ScanComments | scanner.SkipComments - lex.Error = func(*scanner.Scanner, string) {} - depth := 0 - start := true - for token := lex.Scan(); token != scanner.EOF; token = lex.Scan() { - if depth == 0 && start && token == scanner.Ident && lex.TokenText() == "package" { - var parts []string - for { - if lex.Scan() != scanner.Ident { - return "" - } - parts = append(parts, lex.TokenText()) - switch lex.Scan() { - case ';': - return strings.Join(parts, ".") - case '.': - continue - default: - return "" - } - } - } - switch token { - case '{', '(', '[': - depth++ - start = false - case '}', ')', ']': - if depth > 0 { - depth-- - } - start = depth == 0 - case ';': - if depth == 0 { - start = true - } - default: - if depth == 0 { - start = false - } - } - } - return "" + return files, nil } diff --git a/internal/generation/packages_test.go b/internal/generation/packages_test.go index 7a36ff58..1a6de5c4 100644 --- a/internal/generation/packages_test.go +++ b/internal/generation/packages_test.go @@ -9,31 +9,30 @@ import ( v1 "github.com/easyp-tech/easyp/internal/config/v1" ) -func TestSourcePackageReadsOnlyDeclaration(t *testing.T) { +func TestSelectedSourceFilesRespectsBufSelection(t *testing.T) { t.Parallel() - for _, tt := range []struct{ name, source, want string }{ - {name: "simple", source: "syntax = \"proto3\"; package demo.v1; message M {}", want: "demo.v1"}, - {name: "comments", source: "/*package fake;*/ syntax='proto3'; //package wrong;\npackage demo /*comment*/ . v1 ;", want: "demo.v1"}, - {name: "string option", source: "syntax=\"proto3\"; option java_package=\"package wrong;\"; package demo.v1;", want: "demo.v1"}, - {name: "field name", source: "message M { string package = 1; }"}, - {name: "late declaration", source: "message M {} package demo.v1;", want: "demo.v1"}, - {name: "broken unselected body", source: "package other.v1; message {", want: "other.v1"}, - {name: "malformed package", source: "package demo..v1;"}, - {name: "empty", source: ""}, - {name: "byte order mark", source: "\ufeffpackage demo.v1;", want: "demo.v1"}, + for _, tt := range []struct { + name string + paths []string + matchedPaths map[string]bool + }{ + {name: "package_only", matchedPaths: map[string]bool{}}, + {name: "package_and_path", paths: []string{"proto/selected", "proto/excluded"}, matchedPaths: map[string]bool{"proto/selected": true}}, } { - t.Run(tt.name, func(t *testing.T) { t.Parallel(); assert.Equal(t, tt.want, sourcePackage([]byte(tt.source))) }) - } -} + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "proto/selected/a.proto", "syntax = \"proto3\"; package selected.v1; message A {}") + writeV1GenerateFixture(t, root, "proto/excluded/b.proto", "syntax = \"proto3\"; package excluded.v1; message B {}") + selected := v1GenerationModule{directory: root, module: v1.Module{Name: "example.test/dep", Roots: []string{"proto"}, ProtoFilters: []v1.ProtoFileFilter{{Root: "proto", Excludes: []string{"proto/excluded"}}}}} -func TestSelectedPackageFilesRespectsBufSelection(t *testing.T) { - t.Parallel() - root := t.TempDir() - writeV1GenerateFixture(t, root, "proto/selected/a.proto", "syntax = \"proto3\"; package selected.v1; message A {}") - writeV1GenerateFixture(t, root, "proto/excluded/b.proto", "syntax = \"proto3\"; package excluded.v1; message B {}") - selected := v1GenerationModule{directory: root, module: v1.Module{Name: "example.test/dep", Roots: []string{"proto"}, ProtoFilters: []v1.ProtoFileFilter{{Root: "proto", Excludes: []string{"proto/excluded"}}}}} - files, matched, err := selectedPackageFiles(t.Context(), selected, []string{"selected.v1", "excluded.v1"}) - require.NoError(t, err) - assert.Equal(t, []string{"selected/a.proto"}, files) - assert.Equal(t, map[string]bool{"selected.v1": true}, matched) + selectors := newSourceSelectorMatches([]string{"selected.v1", "excluded.v1"}, tt.paths) + files, err := selectedSourceFiles(t.Context(), selected, selectors, newSourceSelectorMatches(nil, nil)) + + require.NoError(t, err) + assert.Equal(t, []string{"selected/a.proto"}, files) + assert.Equal(t, map[string]bool{"selected.v1": true}, selectors.packageMatches) + assert.Equal(t, tt.matchedPaths, selectors.pathMatches) + }) + } } diff --git a/internal/generation/paths_test.go b/internal/generation/paths_test.go new file mode 100644 index 00000000..3f8e36dd --- /dev/null +++ b/internal/generation/paths_test.go @@ -0,0 +1,189 @@ +package generation + +import ( + "fmt" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + v1 "github.com/easyp-tech/easyp/internal/config/v1" + "github.com/easyp-tech/easyp/internal/logger" +) + +func TestRunPathSelectionKeepsRequiredImports(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name string + include bool + }{ + {name: "selected_sources"}, + {name: "include_imports", include: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + files := map[string]string{ + "protobuf.mod": "module example.com/app\nroots proto\n", + "easyp.gen.yaml": "version: v1\ngenerate:\n paths: [proto/api]\n packages: [api.v1]\n", + "proto/api/service.proto": "syntax = \"proto3\"; package api.v1; import \"common/types.proto\"; message Service { common.v1.Type type = 1; }", + "proto/common/types.proto": "syntax = \"proto3\"; package common.v1; message Type {}", + "proto/api-copy/bad.proto": "syntax = \"proto3\"; package api.v1; message Broken {", + "proto/other/bad.proto": "this unrelated source is malformed", + } + for _, directory := range []string{"module-lite", "module-full", "module-kotlin", "module-java"} { + files["proto/examples/jvm/"+directory+"/build/service.proto"] = files["proto/api/service.proto"] + } + for path, content := range files { + writeV1GenerateFixture(t, root, path, content) + } + out := filepath.Join(root, "descriptor.pb") + + require.NoError(t, Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root, DescriptorSetOut: out, IncludeImports: tt.include})) + + want := []string{"api/service.proto"} + if tt.include { + want = append(want, "common/types.proto") + } + assert.ElementsMatch(t, want, descriptorNames(readDescriptorSet(t, out))) + }) + } +} + +func TestRunSourceSelectorsMatchAcrossModules(t *testing.T) { + t.Parallel() + tests := []struct { + name string + packages []string + paths []string + want []string + wantErr string + }{ + {name: "aggregate_each_selector_across_modules", packages: []string{"first.v1", "second.v1"}, paths: []string{"proto/api/first.proto", "proto/api/second.proto"}, want: []string{"api/first.proto", "api/second.proto"}}, + {name: "skip_module_without_combined_match", packages: []string{"first.v1"}, paths: []string{"proto/api"}, want: []string{"api/first.proto"}}, + {name: "unmatched_package_after_path_filter", packages: []string{"first.v1", "second.v1"}, paths: []string{"proto/api/first.proto"}, wantErr: "generate.packages did not match any selected module source files: second.v1"}, + {name: "unmatched_path_after_package_filter", packages: []string{"first.v1"}, paths: []string{"proto/api/first.proto", "proto/api/second.proto"}, wantErr: "generate.paths did not match any selected module source files: proto/api/second.proto"}, + {name: "dot_with_package_filter", packages: []string{"first.v1"}, paths: []string{"."}, want: []string{"api/first.proto"}}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "easyp.gen.yaml", fmt.Sprintf("version: v1\ngenerate:\n modules: [m1, m2]\n packages: [%s]\n paths: [%s]\n", strings.Join(tt.packages, ", "), strings.Join(tt.paths, ", "))) + writeV1GenerateFixture(t, root, "m1/protobuf.mod", "module example.com/first\nroots proto\n") + writeV1GenerateFixture(t, root, "m1/proto/api/first.proto", "syntax = \"proto3\"; package first.v1; message First {}") + writeV1GenerateFixture(t, root, "m2/protobuf.mod", "module example.com/second\nroots proto\n") + writeV1GenerateFixture(t, root, "m2/proto/api/second.proto", "syntax = \"proto3\"; package second.v1; message Second {}") + out := filepath.Join(root, "descriptor.pb") + previous := []byte("previous descriptor") + require.NoError(t, os.WriteFile(out, previous, 0o600)) + + err := Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root, DescriptorSetOut: out}) + + if tt.wantErr != "" { + require.ErrorContains(t, err, tt.wantErr) + actual, readErr := os.ReadFile(out) + require.NoError(t, readErr) + assert.Equal(t, previous, actual) + return + } + require.NoError(t, err) + assert.ElementsMatch(t, tt.want, descriptorNames(readDescriptorSet(t, out))) + }) + } +} + +func TestRunUnknownPathWithoutPlugins(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "easyp.gen.yaml", "version: v1\ngenerate:\n paths: [missing, missing]\n") + writeV1GenerateFixture(t, root, "item.proto", "syntax = \"proto3\"; package item.v1; message Item {}") + + err := Run(t.Context(), logger.NewNop(), nil, Request{WorkDir: root}) + + require.ErrorContains(t, err, "generate.paths did not match any selected module source files: missing") + assert.NotContains(t, err.Error(), "missing, missing") +} + +func TestRunAllValidatesSelectorsOnOptionsOnlyParentBeforeChildren(t *testing.T) { + t.Parallel() + if runtime.GOOS == "windows" { + t.Skip("fixture executable uses a POSIX shell") + } + for _, tt := range []struct { + name string + selector string + export bool + }{ + {name: "paths_without_export", selector: "paths"}, + {name: "paths_with_export", selector: "paths", export: true}, + {name: "packages_without_export", selector: "packages"}, + {name: "packages_with_export", selector: "packages", export: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "easyp.gen.yaml", "version: v1\ngenerate:\n "+tt.selector+": [missing]\noptions:\n go:\n package_prefix: example.com/gen\n") + writeV1GenerateFixture(t, root, "legacy/orphan.proto", "syntax = \"proto3\"; package orphan.v1; message Orphan {}") + writeV1GenerateFixture(t, root, "child/protobuf.mod", "module example.com/child\n") + writeV1GenerateFixture(t, root, "child/easyp.gen.yaml", "version: v1\nplugins:\n - path: ./child-plugin\n out: gen\n") + writeV1GenerateFixture(t, root, "child/item.proto", "syntax = \"proto3\"; package child.v1; message Item {}") + require.NoError(t, os.WriteFile(filepath.Join(root, "child-plugin"), []byte("#!/bin/sh\ncat >/dev/null\nprintf called > child-ran.txt\n"), 0o755)) + request := Request{WorkDir: root, AllProjects: true} + out := filepath.Join(root, "all.pb") + previous := []byte("previous descriptor") + if tt.export { + require.NoError(t, os.WriteFile(out, previous, 0o600)) + request.DescriptorSetOut = out + } + + err := Run(t.Context(), logger.NewNop(), nil, request) + + require.ErrorContains(t, err, "generate."+tt.selector+" did not match any selected module source files: missing") + assert.NoFileExists(t, filepath.Join(root, "child-ran.txt")) + if tt.export { + actual, readErr := os.ReadFile(out) + require.NoError(t, readErr) + assert.Equal(t, previous, actual) + } + }) + } +} + +func TestSelectedSourceFilesSkipsReadingPathExcludedFiles(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "api/item.proto", "syntax = \"proto3\"; package item.v1; message Item {}") + require.NoError(t, os.Symlink(filepath.Join(root, "absent.proto"), filepath.Join(root, "unreadable.proto"))) + selected := v1GenerationModule{directory: root, module: v1.Module{Name: "example.com/app", Roots: []string{"."}}} + + selectors := newSourceSelectorMatches([]string{"item.v1"}, []string{"api"}) + files, err := selectedSourceFiles(t.Context(), selected, selectors, newSourceSelectorMatches(nil, nil)) + + require.NoError(t, err) + assert.Equal(t, []string{"api/item.proto"}, files) + assert.Equal(t, map[string]bool{"item.v1": true}, selectors.packageMatches) + assert.Equal(t, map[string]bool{"api": true}, selectors.pathMatches) +} + +func TestSelectedSourceFilesKeepsSourceBoundaries(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "api/item.proto", "syntax = \"proto3\"; package item.v1; message Item {}") + writeV1GenerateFixture(t, root, ".sources/hidden.proto", "syntax = \"proto3\"; package item.v1; message Hidden {}") + writeV1GenerateFixture(t, root, "nested/protobuf.mod", "module example.com/nested\n") + writeV1GenerateFixture(t, root, "nested/nested.proto", "syntax = \"proto3\"; package item.v1; message Nested {}") + selected := v1GenerationModule{directory: root, module: v1.Module{Name: "example.com/app", Roots: []string{"."}}} + + selectors := newSourceSelectorMatches([]string{"item.v1"}, []string{".", ".sources", "nested"}) + files, err := selectedSourceFiles(t.Context(), selected, selectors, newSourceSelectorMatches(nil, nil)) + + require.NoError(t, err) + assert.Equal(t, []string{"api/item.proto"}, files) + assert.Equal(t, map[string]bool{"item.v1": true}, selectors.packageMatches) + assert.Equal(t, map[string]bool{".": true}, selectors.pathMatches) +} diff --git a/internal/generation/selected_test.go b/internal/generation/selected_test.go index d55888d0..7aaf5ae1 100644 --- a/internal/generation/selected_test.go +++ b/internal/generation/selected_test.go @@ -210,7 +210,7 @@ func TestSelectV1ModulesExplainsMissingLocalModule(t *testing.T) { configDir := filepath.Join(root, "backend") require.NoError(t, os.MkdirAll(configDir, 0o755)) - _, err := selectV1Modules(root, configDir, []string{"."}) + _, err := selectV1Modules(root, configDir, []v1.GenerateModule{{Module: "."}}) require.ErrorContains(t, err, "generate.modules entry \".\" is a local path") require.ErrorContains(t, err, "protobuf.mod") diff --git a/internal/generation/selection.go b/internal/generation/selection.go index 0027e858..95fdda75 100644 --- a/internal/generation/selection.go +++ b/internal/generation/selection.go @@ -13,6 +13,7 @@ import ( v1 "github.com/easyp-tech/easyp/internal/config/v1" "github.com/easyp-tech/easyp/internal/logger" "github.com/easyp-tech/easyp/internal/modules" + "github.com/easyp-tech/easyp/internal/sourceview" "github.com/easyp-tech/easyp/internal/workspace" ) @@ -21,20 +22,25 @@ import ( type v1ModuleSelection struct { directory string source string + index int + packages []string + paths []string } -func selectV1Modules(repoRoot, configDir string, names []string) ([]v1ModuleSelection, error) { - if len(names) == 0 { +func selectV1Modules(repoRoot, configDir string, entries []v1.GenerateModule) ([]v1ModuleSelection, error) { + if len(entries) == 0 { dir, err := generatorV1ModuleDir(repoRoot, configDir) if err != nil { return nil, fmt.Errorf("generatorV1ModuleDir: %w", err) } return []v1ModuleSelection{{directory: dir}}, nil } - result := make([]v1ModuleSelection, 0, len(names)) - for _, name := range names { + result := make([]v1ModuleSelection, 0, len(entries)) + for i, entry := range entries { + name := entry.Module + selection := v1ModuleSelection{source: name, index: i, packages: entry.Packages, paths: entry.Paths} if filepath.IsAbs(name) { - result = append(result, v1ModuleSelection{source: name}) + result = append(result, selection) continue } path := filepath.Clean(filepath.Join(repoRoot, name)) @@ -45,18 +51,19 @@ func selectV1Modules(repoRoot, configDir string, names []string) ([]v1ModuleSele if !filepath.IsLocal(rel) { return nil, fmt.Errorf("module path %q leaves repository", name) } - _, err = os.Stat(filepath.Join(path, v1.ModuleFile)) + _, err = sourceview.ResolveLocal(context.Background(), repoRoot, filepath.Join(rel, v1.ModuleFile)) if errors.Is(err, os.ErrNotExist) { if name == "." || name == ".." || strings.HasPrefix(name, "./") || strings.HasPrefix(name, "../") { return nil, fmt.Errorf("generate.modules entry %q is a local path, but %s has no %s; point to a workspace module directory containing %s, use its module identity, or omit generate.modules to select the module containing this generator", name, path, v1.ModuleFile, v1.ModuleFile) } - result = append(result, v1ModuleSelection{source: name}) + result = append(result, selection) continue } if err != nil { return nil, fmt.Errorf("Stat: %w", err) } - result = append(result, v1ModuleSelection{directory: path}) + selection.directory, selection.source = path, "" + result = append(result, selection) } return result, nil } @@ -195,8 +202,12 @@ func readV1GenerationModule(ctx context.Context, cache modules.Cache, directory func findV1LocalModuleByName(repoRoot, name string) (string, error) { var found string - err := filepath.WalkDir(repoRoot, func(path string, entry fs.DirEntry, walkErr error) error { + err := workspace.Walk(repoRoot, func(path string, entry fs.DirEntry, walkErr error) error { if walkErr != nil { + // Module lookup reads declarations; source aliases are validated after selection. + if filepath.Base(path) != v1.ModuleFile { + return nil + } return walkErr } if entry.IsDir() { @@ -208,7 +219,7 @@ func findV1LocalModuleByName(repoRoot, name string) (string, error) { if entry.Name() != v1.ModuleFile { return nil } - manifest, err := os.ReadFile(path) + manifest, err := workspace.ReadFile(repoRoot, path) if err != nil { return fmt.Errorf("ReadFile: %w", err) } diff --git a/internal/migration/aliases_test.go b/internal/migration/aliases_test.go new file mode 100644 index 00000000..4c319000 --- /dev/null +++ b/internal/migration/aliases_test.go @@ -0,0 +1,161 @@ +package migration + +import ( + "context" + "errors" + "os" + "path/filepath" + "testing" + + "github.com/easyp-tech/easyp/internal/sourceview" + "github.com/stretchr/testify/require" +) + +func TestBuildLinkedMetadataPreservesTarget(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeTransactionFile(t, root, "policy-source.yaml", "lint: {}\n", 0o640) + require.NoError(t, os.Symlink("policy-source.yaml", filepath.Join(root, "easyp.yaml"))) + plan, err := Build(context.Background(), Options{Dir: root, Module: "example.com/api"}) + require.NoError(t, err) + require.NoError(t, plan.Apply()) + assertTransactionFile(t, root, "policy-source.yaml", "lint: {}\n", 0o640) + assertTransactionFile(t, root, "easyp.yaml.v0.bak", "lint: {}\n", 0o640) + info, err := os.Lstat(filepath.Join(root, "easyp.yaml")) + require.NoError(t, err) + require.True(t, info.Mode().IsRegular()) +} + +func TestLinkedMetadataRollbackRestoresPointer(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeTransactionFile(t, root, "policy-source.yaml", "lint: {}\n", 0o640) + require.NoError(t, os.Symlink("policy-source.yaml", filepath.Join(root, "easyp.yaml"))) + plan, err := Build(context.Background(), Options{Dir: root, Module: "example.com/api"}) + require.NoError(t, err) + plan.tx.rename = func(root *os.Root, from, to string) error { + if to == "protobuf.mod" { + return errors.New("injected failure") + } + return root.Rename(from, to) + } + plan.tx.link = func(root *os.Root, from, to string) error { + if to == "protobuf.mod" { + return errors.New("injected failure") + } + return root.Link(from, to) + } + require.ErrorContains(t, plan.Apply(), "injected failure") + pointer, err := os.Readlink(filepath.Join(root, "easyp.yaml")) + require.NoError(t, err) + require.Equal(t, "policy-source.yaml", pointer) + assertTransactionFile(t, root, "policy-source.yaml", "lint: {}\n", 0o640) +} + +func TestLocalSelectionPreservesDirectoryAliasNames(t *testing.T) { + t.Parallel() + root := t.TempDir() + require.NoError(t, os.Mkdir(filepath.Join(root, "sources"), 0o755)) + writeTransactionFile(t, root, "sources/model.proto", "syntax = \"proto3\"; package model;", 0o644) + require.NoError(t, os.Symlink("sources", filepath.Join(root, "proto"))) + require.NoError(t, os.Symlink("missing", filepath.Join(root, "sources", "unused.txt"))) + selection, err := proveLocalSelection(root, []legacyDirectory{{Root: "proto", Path: "."}}, []string{"proto"}) + require.NoError(t, err) + require.Equal(t, map[string]string{"model.proto": "proto/model.proto"}, selection.files) +} + +func TestLinkedInputRechecksPointerAndTarget(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name string + mutate func(*testing.T, string) + }{ + {name: "pointer", mutate: func(t *testing.T, root string) { + writeTransactionFile(t, root, "other-policy.yaml", "lint: {}\n", 0o640) + require.NoError(t, os.Remove(filepath.Join(root, "easyp.yaml"))) + require.NoError(t, os.Symlink("other-policy.yaml", filepath.Join(root, "easyp.yaml"))) + }}, + {name: "target inode", mutate: func(t *testing.T, root string) { + writeTransactionFile(t, root, "other-policy.yaml", "lint: {}\n", 0o640) + require.NoError(t, os.Rename(filepath.Join(root, "other-policy.yaml"), filepath.Join(root, "policy-source.yaml"))) + }}, + {name: "target mode", mutate: func(t *testing.T, root string) { + require.NoError(t, os.Chmod(filepath.Join(root, "policy-source.yaml"), 0o600)) + }}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeTransactionFile(t, root, "policy-source.yaml", "lint: {}\n", 0o640) + require.NoError(t, os.Symlink("policy-source.yaml", filepath.Join(root, "easyp.yaml"))) + plan, err := Build(context.Background(), Options{Dir: root, Module: "example.com/api"}) + require.NoError(t, err) + tt.mutate(t, root) + require.Error(t, plan.Apply()) + require.NoFileExists(t, filepath.Join(root, "protobuf.mod")) + }) + } +} + +func TestNativeLinkedMetadataNoopPreservesTopology(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeTransactionFile(t, root, "policy-source.yaml", "version: v1\nlinters:\n default: MINIMAL\n", 0o640) + writeTransactionFile(t, root, "easyp.gen.yaml", "version: v1\n", 0o644) + writeTransactionFile(t, root, "manifest-source", "module example.com/api\nroots .\n", 0o600) + require.NoError(t, os.Symlink("policy-source.yaml", filepath.Join(root, "easyp.yaml"))) + require.NoError(t, os.Symlink("manifest-source", filepath.Join(root, "protobuf.mod"))) + plan, err := Build(context.Background(), Options{Dir: root, Module: "example.com/api"}) + require.NoError(t, err) + require.True(t, plan.AlreadyV1()) + require.NoError(t, plan.Apply()) + pointer, err := os.Readlink(filepath.Join(root, "easyp.yaml")) + require.NoError(t, err) + require.Equal(t, "policy-source.yaml", pointer) + pointer, err = os.Readlink(filepath.Join(root, "protobuf.mod")) + require.NoError(t, err) + require.Equal(t, "manifest-source", pointer) +} + +func TestBuildAbsoluteInternalSourceAlias(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeTransactionFile(t, root, "easyp.yaml", "generate:\n inputs:\n - directory:\n root: proto\n path: .\n", 0o644) + require.NoError(t, os.Mkdir(filepath.Join(root, "proto"), 0o755)) + writeTransactionFile(t, root, "model-source.proto", "syntax = \"proto3\"; package model.v1; message Model {}", 0o644) + require.NoError(t, os.Symlink(filepath.Join(root, "model-source.proto"), filepath.Join(root, "proto/model.proto"))) + plan, err := Build(context.Background(), Options{Dir: root, Module: "example.com/api"}) + require.NoError(t, err) + require.NoError(t, plan.Apply()) + require.Equal(t, map[string]string{"model.proto": "proto/model.proto"}, plan.sources) +} + +func TestTransactionRejectsOutputOverlappingLinkedInputHop(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeTransactionFile(t, root, "source.proto", "original source", 0o644) + require.NoError(t, os.Symlink("source.proto", filepath.Join(root, "protobuf.mod"))) + require.NoError(t, os.Symlink("protobuf.mod", filepath.Join(root, "alias.proto"))) + tx, err := newTransaction(root) + require.NoError(t, err) + _, err = tx.captureInput("alias.proto") + require.NoError(t, err) + _, err = tx.captureInput("protobuf.mod") + require.NoError(t, err) + tx.changes = []fileChange{{name: "protobuf.mod", content: []byte("replacement"), mode: 0o644}} + require.ErrorContains(t, tx.apply(), "overlaps linked input hop") + pointer, err := os.Readlink(filepath.Join(root, "protobuf.mod")) + require.NoError(t, err) + require.Equal(t, "source.proto", pointer) +} + +func TestBuildSourceRootDirectoryAliasCycleFails(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeTransactionFile(t, root, "easyp.yaml", "generate:\n inputs:\n - directory:\n root: proto\n path: .\n", 0o644) + writeTransactionFile(t, root, "file.proto", "syntax = \"proto3\"; package model.v1; message Model {}", 0o644) + require.NoError(t, os.Mkdir(filepath.Join(root, ".git"), 0o755)) + require.NoError(t, os.Symlink(".", filepath.Join(root, "proto"))) + _, err := Build(context.Background(), Options{Dir: root, Module: "example.com/api"}) + require.ErrorIs(t, err, sourceview.ErrCycle) +} diff --git a/internal/migration/apply.go b/internal/migration/apply.go index 23e5c678..c14e7cb5 100644 --- a/internal/migration/apply.go +++ b/internal/migration/apply.go @@ -1,6 +1,7 @@ package migration import ( + "context" "crypto/rand" "crypto/sha256" "errors" @@ -11,14 +12,20 @@ import ( "path/filepath" "slices" "strings" + + "github.com/easyp-tech/easyp/internal/sourceview" ) type snapshot struct { - name string - data []byte - mode os.FileMode - exists bool - info os.FileInfo + name string + data []byte + mode os.FileMode + exists bool + info os.FileInfo + input bool + resolved string + links []sourceview.Link + pointer string } type fileChange struct { @@ -114,7 +121,7 @@ func (t *transaction) openRoot() (_ *os.Root, resultErr error) { if err != nil { return nil, fmt.Errorf("checkRoot: %w", err) } - root, err := os.OpenRoot(t.root) + root, err := os.OpenRoot(t.requestedRoot) if err != nil { return nil, fmt.Errorf("OpenRoot: %w", err) } @@ -134,6 +141,9 @@ func (t *transaction) openRoot() (_ *os.Root, resultErr error) { } func (t *transaction) capture(name string) (_ snapshot, resultErr error) { + if previous, ok := t.expected[name]; ok && previous.input { + return t.captureInput(name) + } root, err := t.openRoot() if err != nil { return snapshot{}, fmt.Errorf("openRoot: %w", err) @@ -153,6 +163,92 @@ func (t *transaction) capture(name string) (_ snapshot, resultErr error) { return current, nil } +func (t *transaction) captureInput(name string) (_ snapshot, resultErr error) { + root, err := t.openRoot() + if err != nil { + return snapshot{}, fmt.Errorf("openRoot: %w", err) + } + defer func() { resultErr = errors.Join(resultErr, closeRoot(root)) }() + current, err := readInputSnapshot(root, name) + if err != nil { + return snapshot{}, fmt.Errorf("readInputSnapshot: %w", err) + } + if previous, ok := t.expected[name]; ok { + if !sameSnapshot(previous, current) { + return snapshot{}, fmt.Errorf("file %q changed since planning", name) + } + return previous, nil + } + t.expected[name] = current + return current, nil +} + +func readInputSnapshot(root *os.Root, name string) (_ snapshot, resultErr error) { + if err := checkRelativeName(name); err != nil { + return snapshot{}, fmt.Errorf("checkRelativeName: %w", err) + } + canonical, err := filepath.EvalSymlinks(root.Name()) + if err != nil { + return snapshot{}, fmt.Errorf("EvalSymlinks: %w", err) + } + view, err := sourceview.NewLocal(root.FS(), canonical, root.Name()) + if err != nil { + return snapshot{}, fmt.Errorf("NewLocal: %w", err) + } + resolved, err := view.Resolve(context.Background(), filepath.ToSlash(name)) + if errors.Is(err, os.ErrNotExist) && len(resolved.Links) == 0 { + return snapshot{name: name, input: true}, nil + } + if err != nil { + return snapshot{}, fmt.Errorf("Resolve: %w", err) + } + if err := sourceview.CheckLocalResolution(root, resolved); err != nil { + return snapshot{}, fmt.Errorf("CheckLocalResolution: %w", err) + } + if !resolved.Info.Mode().IsRegular() { + return snapshot{}, fmt.Errorf("input %q must resolve to a regular file", name) + } + current, err := readSnapshot(root, filepath.FromSlash(resolved.Path)) + if err != nil { + return snapshot{}, fmt.Errorf("readSnapshot: %w", err) + } + current.name, current.input, current.resolved, current.links = name, true, resolved.Path, resolved.Links + leaf, err := root.Lstat(name) + if err != nil { + return snapshot{}, fmt.Errorf("Lstat: %w", err) + } + if leaf.Mode()&os.ModeSymlink != 0 { + current.pointer, err = root.Readlink(name) + if err != nil { + return snapshot{}, fmt.Errorf("Readlink: %w", err) + } + } + after, err := view.Resolve(context.Background(), filepath.ToSlash(name)) + if err != nil { + return snapshot{}, fmt.Errorf("Resolve: %w", err) + } + if !sameResolution(resolved, after) { + return snapshot{}, fmt.Errorf("input %q changed while reading", name) + } + return current, nil +} + +func sameResolution(before, after sourceview.Resolution) bool { + if before.Path != after.Path || len(before.Links) != len(after.Links) { + return false + } + if !sameFileState(before.Info, after.Info) { + return false + } + for i, link := range before.Links { + other := after.Links[i] + if link.Path != other.Path || link.Target != other.Target || !sameFileState(link.Info, other.Info) { + return false + } + } + return true +} + func readSnapshot(root *os.Root, name string) (_ snapshot, resultErr error) { err := checkRelativeName(name) if err != nil { @@ -216,6 +312,15 @@ func sameSnapshot(before, after snapshot) bool { if before.exists != after.exists { return false } + if before.resolved != after.resolved || before.pointer != after.pointer || len(before.links) != len(after.links) { + return false + } + for i, link := range before.links { + other := after.links[i] + if link.Path != other.Path || link.Target != other.Target || !sameFileState(link.Info, other.Info) { + return false + } + } return !before.exists || (os.SameFile(before.info, after.info) && before.mode == after.mode && sha256.Sum256(before.data) == sha256.Sum256(after.data)) } @@ -261,7 +366,13 @@ func (t *transaction) verify(root *os.Root) error { } func recheckSnapshot(root *os.Root, expected snapshot) error { - current, err := readSnapshot(root, expected.name) + var current snapshot + var err error + if expected.input { + current, err = readInputSnapshot(root, expected.name) + } else { + current, err = readSnapshot(root, expected.name) + } if err != nil { return fmt.Errorf("readSnapshot: %w", err) } @@ -348,6 +459,16 @@ func (t *transaction) validateChanges() error { if _, ok := t.expected[change.name]; !ok { return fmt.Errorf("destination %q was not captured during planning", change.name) } + for _, input := range t.expected { + if len(input.links) > 0 && input.resolved == filepath.ToSlash(change.name) { + return fmt.Errorf("destination %q overlaps linked input target", change.name) + } + for _, link := range input.links { + if link.Path == filepath.ToSlash(change.name) && input.name != change.name { + return fmt.Errorf("destination %q overlaps linked input hop for %q", change.name, input.name) + } + } + } if seen[change.name] { return fmt.Errorf("duplicate destination %q", change.name) } @@ -378,7 +499,11 @@ func (t *transaction) stageChanges(root *os.Root, temporary string) ([]stagedCha item.installed = snapshot{name: change.name, data: change.content, mode: change.mode, exists: true, info: info} before := t.expected[change.name] if before.exists { - err = t.stage(root, item.rollback, before.data, before.mode) + if before.pointer != "" { + err = root.Symlink(before.pointer, item.rollback) + } else { + err = t.stage(root, item.rollback, before.data, before.mode) + } if err != nil { return nil, fmt.Errorf("stage: %w", err) } diff --git a/internal/migration/convert.go b/internal/migration/convert.go index a59210ec..8d7a8981 100644 --- a/internal/migration/convert.go +++ b/internal/migration/convert.go @@ -47,11 +47,13 @@ type generateOutput struct { Version string `yaml:"version"` Generate targetsOutput `yaml:"generate"` Plugins []pluginOutput `yaml:"plugins"` - Options v1.GenerateOptions `yaml:"options"` + Options v1.GenerateOptions `yaml:"options,omitempty"` } type targetsOutput struct { - Modules []string `yaml:"modules,omitempty"` - Managed config.ManagedMode `yaml:"managed,omitempty"` + Modules []string `yaml:"modules,omitempty"` + Packages []string `yaml:"packages,omitempty"` + Paths []string `yaml:"paths,omitempty"` + Managed config.ManagedMode `yaml:"managed,omitempty"` } type pluginOutput struct { Name string `yaml:"name,omitempty"` @@ -230,12 +232,10 @@ func prefixExclusionPaths(value string) ([]string, error) { return []string{prefix, prefix + "/**"}, nil } -func convertGenerate(cfg legacyConfig, selected []string) ([]byte, error) { - // Empty explicitly disables v1 inheritance of a parent's Go package prefix. - emptyPrefix := "" +func convertGenerate(cfg legacyConfig, selected, packages, paths []string) ([]byte, error) { output := generateOutput{ - Version: "v1", Generate: targetsOutput{Modules: selected, Managed: cfg.Generate.Managed}, - Plugins: []pluginOutput{}, Options: v1.GenerateOptions{Go: v1.GoOptions{PackagePrefix: &emptyPrefix}}, + Version: "v1", Generate: targetsOutput{Modules: selected, Packages: packages, Paths: paths, Managed: cfg.Generate.Managed}, + Plugins: []pluginOutput{}, } for i, plugin := range cfg.Generate.Plugins { converted := pluginOutput{Name: plugin.Name, Path: plugin.Path, Command: plugin.Command, diff --git a/internal/migration/empty_path_selection_test.go b/internal/migration/empty_path_selection_test.go new file mode 100644 index 00000000..cb9a88fb --- /dev/null +++ b/internal/migration/empty_path_selection_test.go @@ -0,0 +1,49 @@ +package migration + +import ( + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/easyp-tech/easyp/internal/modules" +) + +func TestMigrationPathsRequireEveryInputToMatch(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name string + outsideCopy bool + }{ + {name: "proven package fallback"}, + {name: "no safe fallback", outsideCopy: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + const dependency = "example.test/dependency" + writeFixture(t, root, "easyp.yaml", "deps: ["+dependency+"@v1.0.0]\ngenerate:\n inputs: [{directory: mcp}, {directory: empty}]\n") + writeFixture(t, root, "mcp/options.proto", "package mcp.options.v1;") + writeFixture(t, root, "empty/.keep", "") + outside := "package other.v1;" + if tt.outsideCopy { + outside = "package mcp.options.v1;" + } + writeFixture(t, root, "outside.proto", outside) + repo := &mockRepository{fetched: map[string]modules.Fetched{dependency + "@v1.0.0": migrationFetched(dependency, "v1.0.0", testCommit)}} + plan, err := Build(t.Context(), Options{Dir: root, Module: "example.test/api", ResolveLock: true, Repository: repo}) + if tt.outsideCopy { + require.ErrorContains(t, err, "scope") + assert.Empty(t, repo.calls, "unsafe source selection must fail before permitted dependency access") + assert.NoFileExists(t, filepath.Join(root, "protobuf.mod")) + return + } + require.NoError(t, err) + assert.Equal(t, []string{dependency + "@v1.0.0"}, repo.calls) + assert.Empty(t, plan.paths, "an empty input cannot become an unmatched runtime selector") + assert.Equal(t, []string{"mcp.options.v1"}, plan.packages) + require.NoError(t, plan.Apply()) + }) + } +} diff --git a/internal/migration/generate_options_test.go b/internal/migration/generate_options_test.go new file mode 100644 index 00000000..5db27815 --- /dev/null +++ b/internal/migration/generate_options_test.go @@ -0,0 +1,48 @@ +package migration + +import ( + "bytes" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "gopkg.in/yaml.v3" + + v1 "github.com/easyp-tech/easyp/internal/config/v1" +) + +func TestMigrationOmitsUnconfiguredGoPrefix(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name string + managed string + enabled bool + }{ + {name: "managed_omitted"}, + {name: "managed_enabled", managed: " managed:\n enabled: true\n override: [{file_option: go_package_prefix, value: example.com/managed}]\n", enabled: true}, + {name: "managed_disabled", managed: " managed:\n enabled: false\n override: [{file_option: go_package_prefix, value: example.com/managed}]\n"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeFixture(t, root, "easyp.yaml", "generate:\n inputs: [{directory: .}]\n plugins: [{name: go, out: gen}]\n"+tt.managed) + writeFixture(t, root, "item.proto", "syntax = \"proto3\"; package item.v1; option go_package = \"example.com/original/item\"; message Item {}") + + plan, err := Build(t.Context(), Options{Dir: root, Module: "example.com/app"}) + require.NoError(t, err) + raw := outputContent(t, plan, v1.GenerateFile) + var document map[string]any + require.NoError(t, yaml.Unmarshal(raw, &document)) + assert.NotContains(t, document, "options", "migration must not manufacture new Go prefix settings") + gen, err := v1.ParseGenerate(bytes.NewReader(raw)) + require.NoError(t, err) + assert.Nil(t, gen.Options.Go.PackagePrefix, "an omitted prefix must remain optional") + assert.Equal(t, tt.enabled, gen.Generate.Managed.Enabled) + if tt.managed != "" { + require.Len(t, gen.Generate.Managed.Override, 1) + assert.Equal(t, "go_package_prefix", gen.Generate.Managed.Override[0].FileOption) + assert.Equal(t, "example.com/managed", gen.Generate.Managed.Override[0].Value) + } + }) + } +} diff --git a/internal/migration/legacy.go b/internal/migration/legacy.go index 4a14a03c..4a5d85fa 100644 --- a/internal/migration/legacy.go +++ b/internal/migration/legacy.go @@ -194,7 +194,7 @@ func typedNode(node *yaml.Node, typ reflect.Type, path string, warnings *[]strin } return fmt.Errorf("null YAML value at line %d", node.Line) } - if typ == reflect.TypeFor[legacyDirectory]() && node.Kind == yaml.ScalarNode && node.Tag == "!!str" { + if (typ == reflect.TypeFor[legacyDirectory]() || typ == reflect.TypeFor[v1.GenerateModule]()) && node.Kind == yaml.ScalarNode && node.Tag == "!!str" { return nil } if typ == reflect.TypeFor[config.PluginOpts]() || typ == reflect.TypeFor[v1.PluginOptions]() { diff --git a/internal/migration/local_module_selection_test.go b/internal/migration/local_module_selection_test.go new file mode 100644 index 00000000..ccbed09f --- /dev/null +++ b/internal/migration/local_module_selection_test.go @@ -0,0 +1,42 @@ +package migration + +import ( + "bytes" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + v1 "github.com/easyp-tech/easyp/internal/config/v1" +) + +func TestMigrationLocalSelectionUsesModuleRelativePaths(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name string + inputs string + files map[string]string + paths []string + }{ + {name: "service_import_root", inputs: "[{directory: {root: api, path: easyp}}]", files: map[string]string{"api/easyp/generator.proto": "package easyp.generator.v1;"}, paths: []string{"api/easyp"}}, + {name: "mixed_roots", inputs: "[{directory: {root: proto, path: .}}, {directory: {root: schema, path: selected}}]", files: map[string]string{"proto/first.proto": "package first.v1;", "schema/selected/second.proto": "package second.v1;", "schema/other.proto": "package other.v1;"}, paths: []string{"proto", "schema/selected"}}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeFixture(t, root, "easyp.yaml", "generate:\n inputs: "+tt.inputs+"\n") + for name, content := range tt.files { + writeFixture(t, root, name, content) + } + + plan, err := Build(t.Context(), Options{Dir: root, Module: "example.com/service"}) + require.NoError(t, err) + gen, err := v1.ParseGenerate(bytes.NewReader(outputContent(t, plan, v1.GenerateFile))) + require.NoError(t, err) + assert.Empty(t, gen.Generate.Modules, "the generator must infer its sole local module") + assert.Empty(t, gen.Generate.Packages, "literal physical paths must preserve mixed-root input scope") + assert.Equal(t, tt.paths, gen.Generate.Paths) + assert.Nil(t, gen.Options.Go.PackagePrefix) + }) + } +} diff --git a/internal/migration/lock.go b/internal/migration/lock.go index 9ff14465..1a4aa313 100644 --- a/internal/migration/lock.go +++ b/internal/migration/lock.go @@ -39,7 +39,14 @@ func parseLegacyLock(raw []byte) (map[string]legacyPin, error) { if err := validIdentity(fields[0]); err != nil { return nil, fmt.Errorf("validIdentity: %w", err) } - version, err := migrationVersion(fields[1]) + legacyVersion := fields[1] + // v0 could persist the peeled-ref line from an annotated Git tag. + // Only full SemVer tags qualify; pseudo-shaped names must retain their + // tag semantics rather than entering legacy pseudo-to-commit conversion. + if tag, peeled := strings.CutSuffix(legacyVersion, "^{}"); peeled && semver.IsValid(tag) && fullSemver.MatchString(tag) && !legacyPseudo.MatchString(tag) { + legacyVersion = tag + } + version, err := migrationVersion(legacyVersion) if err != nil { return nil, fmt.Errorf("migrationVersion: %w", err) } diff --git a/internal/migration/manifest_test.go b/internal/migration/manifest_test.go index 1f30e198..6f0d4331 100644 --- a/internal/migration/manifest_test.go +++ b/internal/migration/manifest_test.go @@ -32,7 +32,7 @@ func TestManifestBackupAndCombinedDependencies(t *testing.T) { require.NoError(t, err) var generation v1.Generate require.NoError(t, yaml.Unmarshal(outputContent(t, plan, "easyp.gen.yaml"), &generation)) - assert.Equal(t, []string{"example.com/acme/c"}, generation.Generate.Modules) + assert.Equal(t, []v1.GenerateModule{{Module: "example.com/acme/c"}}, generation.Generate.Modules) candidate := string(outputContent(t, plan, "protobuf.mod")) assert.Contains(t, candidate, "require example.com/acme/a v1.0.0") assert.Contains(t, candidate, "require example.com/acme/b v1.0.0 // indirect") @@ -54,7 +54,7 @@ func TestDirectoryPathIsRelativeToRoot(t *testing.T) { t.Parallel() for _, tt := range []struct{ name, extra, wantError string }{ {name: "same_import_names"}, - {name: "scope_widening", extra: "proto/other.proto", wantError: "scope"}, + {name: "path_selection_preserves_scope", extra: "proto/other.proto"}, {name: "hidden_scope_narrowing", extra: "proto/api/.hidden/extra.proto", wantError: "scope"}, } { t.Run(tt.name, func(t *testing.T) { @@ -73,6 +73,7 @@ func TestDirectoryPathIsRelativeToRoot(t *testing.T) { require.NoError(t, err) assert.Equal(t, map[string]string{"api/a.proto": filepath.Join("proto", "api", "a.proto")}, plan.sources) assert.Equal(t, []string{"proto"}, plan.roots) + assert.Equal(t, []string{"proto/api"}, plan.paths) }) } } diff --git a/internal/migration/migration.go b/internal/migration/migration.go index a4616485..043eca12 100644 --- a/internal/migration/migration.go +++ b/internal/migration/migration.go @@ -56,6 +56,8 @@ type Plan struct { alreadyV1 bool inputs []legacyDirectory roots []string + packages []string + paths []string sources map[string]string } @@ -116,11 +118,11 @@ func (p *Plan) Apply() error { func (p *Plan) verifySourceSelection() error { if len(p.inputs) > 0 { - sources, err := proveLocalSelection(p.tx.root, p.inputs, p.roots) + selection, err := proveLocalSelection(p.tx.requestedRoot, p.inputs, p.roots) if err != nil { return fmt.Errorf("proveLocalSelection: %w", err) } - if !maps.Equal(sources, p.sources) { + if !maps.Equal(selection.files, p.sources) || !slices.Equal(selection.packages, p.packages) || !slices.Equal(selection.paths, p.paths) { return fmt.Errorf("local .proto source selection changed since planning; preview again") } } @@ -142,7 +144,11 @@ func Build(ctx context.Context, options Options) (*Plan, error) { } p := &Plan{tx: tx} for _, name := range []string{v1.PolicyFile, v1.GenerateFile, v1.ModuleFile, v1.LockFile, "easyp.lock", "easyp.yaml.v0.bak", "protobuf.mod.v0.bak"} { - if _, err := tx.capture(name); err != nil { + capture := tx.captureInput + if strings.HasSuffix(name, ".v0.bak") { + capture = tx.capture + } + if _, err := capture(name); err != nil { return nil, fmt.Errorf("capture: %w", err) } } @@ -176,12 +182,25 @@ func Build(ctx context.Context, options Options) (*Plan, error) { if err := checkManagedLocal(cfg, options.Module, len(inputs) > 0); err != nil { return nil, fmt.Errorf("checkManagedLocal: %w", err) } - p.sources, err = proveLocalSelection(tx.root, inputs, roots) + selection, err := proveLocalSelection(tx.requestedRoot, inputs, roots) if err != nil { return nil, fmt.Errorf("proveLocalSelection: %w", err) } + p.sources, p.packages, p.paths = selection.files, selection.packages, selection.paths + if len(p.packages) > 0 || len(p.paths) > 0 { + for _, input := range cfg.Generate.Inputs { + if input.GitRepo != nil { + return nil, fmt.Errorf("inferred local selectors would change the generation scope of whole-module Git inputs; migrate separate generation projects manually") + } + } + if len(p.paths) > 0 { + p.warnings = append(p.warnings, "Legacy directory selection is preserved through literal generate.paths selectors relative to the module directory. Import roots and source paths stay unchanged; files outside these paths do not become targets even when they declare the same package.") + } else { + p.warnings = append(p.warnings, "Legacy directory selection is preserved through exact generate.packages selectors. Import roots and source paths stay unchanged; future files declaring those packages also participate in generation.") + } + } for _, source := range p.sources { - if _, err := tx.capture(source); err != nil { + if _, err := tx.captureInput(source); err != nil { return nil, fmt.Errorf("capture: %w", err) } } @@ -211,6 +230,11 @@ func Build(ctx context.Context, options Options) (*Plan, error) { selected = append(selected, name) } } + // The generator already belongs to this module. Keep explicit selections + // only when legacy Git inputs add other generation modules. + if len(inputs) > 0 && len(selected) == 1 && selected[0] == options.Module { + selected = nil + } manifest := tx.expected[v1.ModuleFile] nativeManifest := manifest.exists && v1.IsModuleManifest(manifest.data) var replacements []v1.Replacement @@ -241,7 +265,7 @@ func Build(ctx context.Context, options Options) (*Plan, error) { if err != nil { return nil, fmt.Errorf("convertPolicy: %w", err) } - generateBytes, err := convertGenerate(cfg, selected) + generateBytes, err := convertGenerate(cfg, selected, p.packages, p.paths) if err != nil { return nil, fmt.Errorf("convertGenerate: %w", err) } @@ -323,6 +347,9 @@ func (p *Plan) addOutput(name string, content []byte, replaceLegacy bool) error if err != nil { return fmt.Errorf("capture: %w", err) } + if len(current.links) > 0 && !replaceLegacy && !bytes.Equal(current.data, content) { + return fmt.Errorf("destination %q must be a regular file", name) + } mode := os.FileMode(0o644) if current.exists { mode = current.mode diff --git a/internal/migration/migration_test.go b/internal/migration/migration_test.go index 8e3e46cd..5b4f0b6f 100644 --- a/internal/migration/migration_test.go +++ b/internal/migration/migration_test.go @@ -129,7 +129,7 @@ generate: assert.Equal(t, "v1.2.3", gen.Plugins[1].Version) assert.Equal(t, "${GO_PREFIX}", gen.Generate.Managed.Override[0].Value) assert.Equal(t, "foo", gen.Generate.Managed.Override[0].Path) - assert.Equal(t, []string{"example.com/acme/api"}, gen.Generate.Modules) + assert.Empty(t, gen.Generate.Modules) module, err := v1.ParseModule(bytes.NewReader(outputContent(t, plan, "protobuf.mod"))) require.NoError(t, err) assert.Equal(t, []string{"proto"}, module.Roots) @@ -144,7 +144,7 @@ func TestRejectedInputsDoNotWrite(t *testing.T) { {name: "empty", input: "", want: "empty"}, {name: "alias", input: "lint: &lint {}\nbreaking: *lint\n", want: "alias"}, {name: "duplicate", input: "lint: {}\nlint: {}\n", want: "duplicate"}, - {name: "sliced", input: "generate:\n inputs: [{directory: {root: ., path: selected}}]\n", want: "scope"}, + {name: "nested_boundary", input: "generate:\n inputs: [{directory: {root: ., path: selected}}]\n", want: "scope"}, {name: "external", input: "generate:\n inputs: [{directory: {root: ../external, path: .}}]\n", want: "manual"}, {name: "placeholder", input: "generate:\n inputs: [{directory: '${ROOT}'}]\n", want: "placeholder"}, {name: "git_slice", input: "generate:\n inputs: [{git_repo: {url: example.com/acme/deps, sub_directory: api}}]\n", want: "manual"}, @@ -157,6 +157,9 @@ func TestRejectedInputsDoNotWrite(t *testing.T) { writeFixture(t, root, "easyp.yaml", tt.input) writeFixture(t, root, "selected/a.proto", "syntax = \"proto3\";\n") writeFixture(t, root, "other.proto", "syntax = \"proto3\";\n") + if tt.name == "nested_boundary" { + writeFixture(t, root, "selected/protobuf.mod", "module example.test/nested\n") + } _, err := Build(context.Background(), Options{Dir: root, Module: "example.com/acme/api"}) require.ErrorContains(t, err, tt.want) assert.Equal(t, tt.input, string(mustRead(t, root, "easyp.yaml"))) diff --git a/internal/migration/package_selection_test.go b/internal/migration/package_selection_test.go new file mode 100644 index 00000000..1012208c --- /dev/null +++ b/internal/migration/package_selection_test.go @@ -0,0 +1,213 @@ +package migration + +import ( + "bytes" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + v1 "github.com/easyp-tech/easyp/internal/config/v1" +) + +func TestMigrationPackageSelectionPreservesLegacyInputs(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name, inputs string + files map[string]string + wantPaths []string + wantFiles map[string]string + }{ + { + name: "explicit default root", + inputs: "[{directory: {path: mcp, root: .}}]", + files: map[string]string{ + "mcp/options/v1/options.proto": "syntax = \"proto3\"; package mcp.options.v1; message Options {}", + "internal/testproto/example.proto": "syntax = \"proto3\"; package example.v1; message Example {}", + "testdata/unsupported.proto": "package unsupported.v1; message {", + }, + wantPaths: []string{"mcp"}, + wantFiles: map[string]string{"mcp/options/v1/options.proto": "mcp/options/v1/options.proto"}, + }, + { + name: "omitted root defaults to dot", + inputs: "[{directory: {path: mcp}}]", + files: map[string]string{"mcp/a.proto": "package mcp.v1;", "other.proto": "package other.v1;"}, + wantPaths: []string{"mcp"}, wantFiles: map[string]string{"mcp/a.proto": "mcp/a.proto"}, + }, + { + name: "empty root defaults to dot", + inputs: "[{directory: {path: mcp, root: ''}}]", + files: map[string]string{"mcp/a.proto": "package mcp.v1;", "other.proto": "package other.v1;"}, + wantPaths: []string{"mcp"}, wantFiles: map[string]string{"mcp/a.proto": "mcp/a.proto"}, + }, + { + name: "complete packages and overlapping inputs", + inputs: "[{directory: {path: selected, root: .}}, {directory: {path: selected/a, root: .}}]", + files: map[string]string{ + "selected/a/first.proto": "/* package fake; */ package z.v1;", + "selected/a/second.proto": "package z.v1;", + "selected/b/third.proto": "option java_package = \"package fake;\"; package a.v1;", + "other.proto": "package other.v1;", + }, + wantPaths: []string{"selected", "selected/a"}, + wantFiles: map[string]string{"selected/a/first.proto": "selected/a/first.proto", "selected/a/second.proto": "selected/a/second.proto", "selected/b/third.proto": "selected/b/third.proto"}, + }, + { + name: "whole root keeps selection omitted", + inputs: "[{directory: .}]", + files: map[string]string{"api.proto": "syntax = \"proto3\";"}, + wantFiles: map[string]string{"api.proto": "api.proto"}, + }, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + legacy := "generate:\n inputs: " + tt.inputs + "\n plugins: [{name: go, out: ., opts: {paths: source_relative}}]\n" + writeFixture(t, root, "easyp.yaml", legacy) + for name, content := range tt.files { + writeFixture(t, root, name, content) + } + plan, err := Build(t.Context(), Options{Dir: root, Module: "example.com/acme/api"}) + require.NoError(t, err) + assert.Equal(t, tt.wantFiles, plan.sources) + gen, err := v1.ParseGenerate(bytes.NewReader(outputContent(t, plan, v1.GenerateFile))) + require.NoError(t, err) + assert.Empty(t, gen.Generate.Packages) + assert.Equal(t, tt.wantPaths, gen.Generate.Paths) + require.Len(t, gen.Plugins, 1) + assert.Equal(t, ".", gen.Plugins[0].Out) + assert.Equal(t, []string{"source_relative"}, gen.Plugins[0].Opts["paths"]) + module, err := v1.ParseModule(bytes.NewReader(outputContent(t, plan, v1.ModuleFile))) + require.NoError(t, err) + assert.Equal(t, []string{"."}, module.Roots) + require.NoError(t, plan.Apply()) + assert.Equal(t, legacy, string(mustRead(t, root, "easyp.yaml.v0.bak"))) + for name, content := range tt.files { + assert.Equal(t, content, string(mustRead(t, root, name)), "source %s must stay at its original path", name) + } + }) + } +} + +func TestMigrationPackageSelectionRejectsChangedScope(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name, path, selected, other, extraInput string + files map[string]string + }{ + {name: "hidden source", path: ".selected", selected: "package selected.v1;", other: "package other.v1;"}, + {name: "vendor source", path: "easyp_vendor", selected: "package selected.v1;", other: "package other.v1;"}, + {name: "nested module", path: "selected", selected: "package selected.v1;", other: "package other.v1;", files: map[string]string{"selected/protobuf.mod": "module example.com/nested\n"}}, + {name: "mixed whole Git input", path: "selected", selected: "package selected.v1;", other: "package other.v1;", extraInput: ", {git_repo: {url: example.com/acme/dependency}}"}, + {name: "empty subtree cannot select no packages", path: "selected", other: "package other.v1;"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + legacy := "generate:\n inputs: [{directory: {path: " + tt.path + ", root: .}}" + tt.extraInput + "]\n" + writeFixture(t, root, "easyp.yaml", legacy) + if tt.selected == "" { + writeFixture(t, root, filepath.Join(tt.path, ".keep"), "") + } else { + writeFixture(t, root, filepath.Join(tt.path, "a.proto"), tt.selected) + } + writeFixture(t, root, "other.proto", tt.other) + for name, content := range tt.files { + writeFixture(t, root, name, content) + } + repo := &mockRepository{} + _, err := Build(t.Context(), Options{Dir: root, Module: "example.com/acme/api", ResolveLock: true, Repository: repo}) + require.ErrorContains(t, err, "scope") + assert.Empty(t, repo.calls, "scope must be checked before dependency access") + assert.Equal(t, legacy, string(mustRead(t, root, "easyp.yaml"))) + for _, name := range []string{v1.GenerateFile, v1.ModuleFile, v1.LockFile, "easyp.yaml.v0.bak"} { + _, err := os.Stat(filepath.Join(root, name)) + assert.ErrorIs(t, err, os.ErrNotExist) + } + }) + } +} + +func TestMigrationPackageSelectionRejectsCollidingImportRoots(t *testing.T) { + t.Parallel() + root := t.TempDir() + legacy := "generate:\n inputs: [{directory: {path: ., root: a}}, {directory: {path: ., root: b}}]\n" + writeFixture(t, root, "easyp.yaml", legacy) + writeFixture(t, root, "a/same.proto", "package first.v1;") + writeFixture(t, root, "b/same.proto", "package second.v1;") + repo := &mockRepository{} + _, err := Build(t.Context(), Options{Dir: root, Module: "example.com/acme/api", ResolveLock: true, Repository: repo}) + require.ErrorContains(t, err, "roots collide") + assert.Empty(t, repo.calls) + assert.Equal(t, legacy, string(mustRead(t, root, "easyp.yaml"))) + assert.NoFileExists(t, filepath.Join(root, v1.ModuleFile)) +} + +func TestMigrationPackageSelectionRechecksFixedSelectors(t *testing.T) { + t.Parallel() + root := t.TempDir() + legacy := "generate:\n inputs: [{directory: {path: selected}}]\n" + writeFixture(t, root, "easyp.yaml", legacy) + writeFixture(t, root, "selected/a.proto", "package selected.v1;") + writeFixture(t, root, "other.proto", "package other.v1;") + plan, err := Build(t.Context(), Options{Dir: root, Module: "example.com/acme/api"}) + require.NoError(t, err) + require.Equal(t, []string{"selected"}, plan.paths) + require.NoError(t, os.Remove(filepath.Join(root, "other.proto"))) + recomputed, err := proveLocalSelection(root, plan.inputs, plan.roots) + require.NoError(t, err) + require.Equal(t, plan.sources, recomputed.files, "selected paths and bytes are unchanged") + require.Equal(t, plan.paths, recomputed.paths, "removing outside files must not drop the approved directory filter") + require.NoError(t, plan.CheckUnchanged()) + require.NoError(t, plan.Apply()) + assert.Equal(t, legacy, string(mustRead(t, root, "easyp.yaml.v0.bak"))) +} + +func TestMigrationPackageSelectionRechecksInitiallyEmptyTree(t *testing.T) { + t.Parallel() + root := t.TempDir() + legacy := "generate:\n inputs: [{directory: {path: selected}}]\n" + writeFixture(t, root, "easyp.yaml", legacy) + writeFixture(t, root, "selected/.keep", "") + plan, err := Build(t.Context(), Options{Dir: root, Module: "example.com/acme/api"}) + require.NoError(t, err) + writeFixture(t, root, "other.proto", "package other.v1;") + require.Error(t, plan.CheckUnchanged()) + require.Error(t, plan.Apply()) + assert.Equal(t, legacy, string(mustRead(t, root, "easyp.yaml"))) + assert.NoFileExists(t, filepath.Join(root, v1.ModuleFile)) +} + +func TestMigrationPackageSelectionRechecksUnselectedDeclarations(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name, changed string + }{ + {name: "new selected source", changed: "selected/new.proto"}, + {name: "selected package changes", changed: "selected/a.proto"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + legacy := "generate:\n inputs: [{directory: {path: selected}}]\n" + writeFixture(t, root, "easyp.yaml", legacy) + writeFixture(t, root, "selected/a.proto", "package selected.v1;") + writeFixture(t, root, "other.proto", "package other.v1;") + plan, err := Build(t.Context(), Options{Dir: root, Module: "example.com/acme/api"}) + require.NoError(t, err) + content := "package selected.v1; message Added {}" + if tt.changed == "selected/a.proto" { + content = "package renamed.v1;" + } + writeFixture(t, root, tt.changed, content) + require.Error(t, plan.CheckUnchanged()) + require.Error(t, plan.Apply()) + assert.Equal(t, legacy, string(mustRead(t, root, "easyp.yaml"))) + _, err = os.Stat(filepath.Join(root, v1.ModuleFile)) + assert.ErrorIs(t, err, os.ErrNotExist) + }) + } +} diff --git a/internal/migration/path_fallback_test.go b/internal/migration/path_fallback_test.go new file mode 100644 index 00000000..635707a9 --- /dev/null +++ b/internal/migration/path_fallback_test.go @@ -0,0 +1,46 @@ +package migration + +import ( + "path/filepath" + "runtime" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestMigrationNonportablePathCandidateUsesStrictPackageFallback(t *testing.T) { + t.Parallel() + if runtime.GOOS == "windows" { + t.Skip("the legacy colon directory cannot be created on Windows") + } + for _, tt := range []struct { + name string + partial bool + }{ + {name: "complete package"}, + {name: "partial package rejects fallback", partial: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeFixture(t, root, "easyp.yaml", "generate:\n inputs: [{directory: 'api:legacy'}]\n") + writeFixture(t, root, "api:legacy/a.proto", "package selected.v1;") + outside := "package other.v1;" + if tt.partial { + outside = "package selected.v1;" + } + writeFixture(t, root, "other.proto", outside) + plan, err := Build(t.Context(), Options{Dir: root, Module: "example.test/api"}) + if tt.partial { + require.ErrorContains(t, err, "scope") + assert.NoFileExists(t, filepath.Join(root, "protobuf.mod")) + return + } + require.NoError(t, err) + assert.Empty(t, plan.paths) + assert.Equal(t, []string{"selected.v1"}, plan.packages) + require.NoError(t, plan.Apply()) + }) + } +} diff --git a/internal/migration/path_selection_test.go b/internal/migration/path_selection_test.go new file mode 100644 index 00000000..bf5037d0 --- /dev/null +++ b/internal/migration/path_selection_test.go @@ -0,0 +1,99 @@ +package migration + +import ( + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "gopkg.in/yaml.v3" + + v1 "github.com/easyp-tech/easyp/internal/config/v1" +) + +func TestMigrationPathsPreserveGradleCopies(t *testing.T) { + t.Parallel() + root := t.TempDir() + legacy := "generate:\n inputs: [{directory: {path: mcp, root: .}}]\n plugins: [{name: go, out: ., opts: {paths: source_relative}}]\n" + writeFixture(t, root, "easyp.yaml", legacy) + writeFixture(t, root, ".gitignore", "examples/jvm/*/build/\n") + const source = "syntax = \"proto3\"; package mcp.options.v1; message Options {}\n" + files := []string{"mcp/options/v1/options.proto"} + for _, module := range []string{"java-server", "kotlin-server"} { + for _, kind := range []string{"resources/main", "staged-proto"} { + files = append(files, "examples/jvm/"+module+"/build/"+kind+"/mcp/options/v1/options.proto") + } + } + for _, name := range files { + writeFixture(t, root, name, source) + } + writeFixture(t, root, "other.proto", "package other.v1; message {\n") + plan, err := Build(t.Context(), Options{Dir: root, Module: "example.com/sdk"}) + require.NoError(t, err) + assert.Equal(t, map[string]string{"mcp/options/v1/options.proto": "mcp/options/v1/options.proto"}, plan.sources) + var output struct { + Generate struct { + Paths []string + Packages []string + } + } + require.NoError(t, yaml.Unmarshal(outputContent(t, plan, v1.GenerateFile), &output)) + assert.Equal(t, []string{"mcp"}, output.Generate.Paths) + assert.Empty(t, output.Generate.Packages) + require.NoError(t, plan.Apply()) + assert.Equal(t, legacy, string(mustRead(t, root, "easyp.yaml.v0.bak"))) + for _, name := range files { + assert.Equal(t, source, string(mustRead(t, root, name))) + } +} + +func TestMigrationPathsStayFixedWhenOutsideCopyAppears(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeFixture(t, root, "easyp.yaml", "generate:\n inputs: [{directory: {path: mcp}}]\n") + writeFixture(t, root, "mcp/options.proto", "package mcp.options.v1;") + plan, err := Build(t.Context(), Options{Dir: root, Module: "example.com/sdk"}) + require.NoError(t, err) + require.Equal(t, []string{"mcp"}, plan.paths, "a clean tree must retain literal input scope for future builds") + writeFixture(t, root, "build/copied.proto", "package mcp.options.v1;") + require.NoError(t, plan.CheckUnchanged()) + require.NoError(t, plan.Apply()) + assert.FileExists(t, filepath.Join(root, "build/copied.proto")) +} + +func TestMigrationPathsPreservePartialOrUndeclaredPackages(t *testing.T) { + t.Parallel() + for _, tt := range []struct{ name, content string }{ + {name: "same package outside targets", content: "package shared.v1; message Selected {}"}, + {name: "no package declaration", content: "syntax = \"proto3\"; message Selected {}"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeFixture(t, root, "easyp.yaml", "generate:\n inputs: [{directory: selected}]\n") + writeFixture(t, root, "selected/a.proto", tt.content) + writeFixture(t, root, "other.proto", "package shared.v1; message Other {}") + plan, err := Build(t.Context(), Options{Dir: root, Module: "example.test/api"}) + require.NoError(t, err) + assert.Equal(t, []string{"selected"}, plan.paths) + assert.Equal(t, map[string]string{"selected/a.proto": "selected/a.proto"}, plan.sources) + require.NoError(t, plan.Apply()) + assert.Equal(t, tt.content, string(mustRead(t, root, "selected/a.proto"))) + }) + } +} + +func TestMigrationPathsPreserveMixedRootSelection(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeFixture(t, root, "easyp.yaml", "generate:\n inputs: [{directory: {path: ., root: a}}, {directory: {path: selected, root: b}}]\n") + writeFixture(t, root, "a/first.proto", "package first.v1;") + writeFixture(t, root, "b/selected/second.proto", "package second.v1;") + writeFixture(t, root, "b/outside/other.proto", "package other.v1;") + plan, err := Build(t.Context(), Options{Dir: root, Module: "example.test/api"}) + require.NoError(t, err) + assert.Equal(t, []string{"a", "b/selected"}, plan.paths) + assert.Empty(t, plan.packages, "module-relative paths retain each root's directory selection") + assert.Equal(t, map[string]string{"first.proto": "a/first.proto", "selected/second.proto": "b/selected/second.proto"}, plan.sources) + require.NoError(t, plan.Apply()) +} diff --git a/internal/migration/peeled_lock_test.go b/internal/migration/peeled_lock_test.go new file mode 100644 index 00000000..7b995b4e --- /dev/null +++ b/internal/migration/peeled_lock_test.go @@ -0,0 +1,77 @@ +package migration + +import ( + "errors" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/easyp-tech/easyp/internal/modules" +) + +func TestMigrationPeeledLockPreservesHistoricalHashAndBackup(t *testing.T) { + t.Parallel() + root := t.TempDir() + const dependency = "example.com/acme/dependency" + legacyLock := dependency + " v0.4.0^{} " + testHash + "\n" + writeFixture(t, root, "easyp.yaml", "deps: ["+dependency+"]\n") + writeFixture(t, root, "easyp.lock", legacyLock) + preview, err := Build(t.Context(), Options{Dir: root, Module: "example.com/acme/api"}) + require.NoError(t, err) + assert.True(t, preview.NeedsLockResolution()) + require.Error(t, preview.Apply()) + repo := &mockRepository{ + fetched: map[string]modules.Fetched{dependency + "@v0.4.0": migrationFetched(dependency, "v0.4.0", testCommit)}, + wantOldHash: testHash, + } + plan, err := Build(t.Context(), Options{Dir: root, Module: "example.com/acme/api", ResolveLock: true, Repository: repo}) + require.NoError(t, err) + require.NoError(t, plan.Apply()) + assert.Equal(t, []string{dependency + "@v0.4.0"}, repo.calls) + assert.Equal(t, legacyLock, string(mustRead(t, root, "easyp.lock"))) + lock, err := validateNativeLock(mustRead(t, root, "protobuf.lock")) + require.NoError(t, err) + require.Len(t, lock.Modules, 1) + assert.Equal(t, "v0.4.0", lock.Modules[0].Version) + assert.Equal(t, testCommit, lock.Modules[0].Commit) + assert.Equal(t, testNewHash, lock.Modules[0].Hash) +} + +func TestMigrationPeeledLockRejectsUnsupportedPins(t *testing.T) { + t.Parallel() + for _, tt := range []struct{ name, version string }{ + {name: "branch", version: "main^{}"}, + {name: "short semver", version: "v1.2^{}"}, + {name: "commit expression", version: testCommit + "^{}"}, + {name: "double expression", version: "v1.2.3^{}^{}"}, + {name: "peeled pseudo-shaped tag", version: "v0.0.0-20260101123456-" + testCommit + "^{}"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + _, err := parseLegacyLock([]byte("example.com/acme/dependency " + tt.version + " " + testHash + "\n")) + require.ErrorContains(t, err, "unsupported ref") + }) + } +} + +func TestMigrationPeeledLockPreservesVerificationError(t *testing.T) { + t.Parallel() + root := t.TempDir() + const dependency = "example.com/acme/dependency" + legacy := "deps: [" + dependency + "]\n" + legacyLock := dependency + " v0.4.0^{} " + testHash + "\n" + writeFixture(t, root, "easyp.yaml", legacy) + writeFixture(t, root, "easyp.lock", legacyLock) + mismatch := errors.New("legacy hash mismatch") + repo := &mockRepository{err: mismatch} + _, err := Build(t.Context(), Options{Dir: root, Module: "example.com/acme/api", ResolveLock: true, Repository: repo}) + require.ErrorIs(t, err, mismatch) + assert.Equal(t, []string{dependency + "@v0.4.0"}, repo.calls) + assert.Equal(t, legacyLock, string(mustRead(t, root, "easyp.lock"))) + assert.Equal(t, legacy, string(mustRead(t, root, "easyp.yaml"))) + for _, name := range []string{"protobuf.mod", "protobuf.lock", "easyp.gen.yaml", "easyp.yaml.v0.bak"} { + assert.NoFileExists(t, filepath.Join(root, name)) + } +} diff --git a/internal/migration/roots.go b/internal/migration/roots.go index 117928f0..cb3499c1 100644 --- a/internal/migration/roots.go +++ b/internal/migration/roots.go @@ -1,15 +1,19 @@ package migration import ( + "context" + "errors" "fmt" "io/fs" "maps" - "os" "path/filepath" "slices" "strings" + v1 "github.com/easyp-tech/easyp/internal/config/v1" "github.com/easyp-tech/easyp/internal/core/path_helpers" + "github.com/easyp-tech/easyp/internal/protosource" + "github.com/easyp-tech/easyp/internal/sourceview" ) func localRoots(cfg legacyConfig) ([]string, []legacyDirectory, error) { @@ -47,83 +51,206 @@ func localRoots(cfg legacyConfig) ([]string, []legacyDirectory, error) { return roots, inputs, nil } +type localSourceSelection struct { + files map[string]string + packages []string + paths []string +} + +const localSelectionScopeError = "v1 roots would change generation scope or .proto import names (including hidden/vendor/nested modules); whole roots, exact paths and complete protobuf packages cannot preserve this configuration; split the module or migrate separate generation projects manually" + // proveLocalSelection compares both the physical files and their import names. -// A subset can only migrate when whole v1 roots select exactly the same files. -func proveLocalSelection(root string, inputs []legacyDirectory, roots []string) (map[string]string, error) { +// Paths preserve literal directory inputs, including future files in those +// directories. Complete packages are a fallback for mixed-root selections. +func proveLocalSelection(root string, inputs []legacyDirectory, roots []string) (localSourceSelection, error) { legacy, current := make(map[string]string), make(map[string]string) for _, input := range inputs { importRoot := filepath.Join(root, input.Root) search := filepath.Join(importRoot, input.Path) if err := collectProto(root, importRoot, search, false, legacy); err != nil { - return nil, fmt.Errorf("collectProto: %w", err) + return localSourceSelection{}, fmt.Errorf("collectProto: %w", err) } } for _, moduleRoot := range roots { importRoot := filepath.Join(root, moduleRoot) if err := collectProto(root, importRoot, importRoot, true, current); err != nil { - return nil, fmt.Errorf("collectProto: %w", err) + return localSourceSelection{}, fmt.Errorf("collectProto: %w", err) + } + } + wholeRoots := make(map[string]bool) + for _, input := range inputs { + if input.Path == "." { + wholeRoots[input.Root] = true } } - if !maps.Equal(legacy, current) { - return nil, fmt.Errorf("v1 roots would change generation scope or .proto import names (including hidden/vendor/nested modules); split the module or reorganize legacy inputs manually; no v1 directory selection syntax can preserve this configuration") + allRoots := true + for _, moduleRoot := range roots { + allRoots = allRoots && wholeRoots[moduleRoot] + } + // A directory subset must retain its path even when no outside source + // exists yet; otherwise a later build copy would silently widen targets. + if maps.Equal(legacy, current) && (allRoots || len(legacy) == 0) { + return localSourceSelection{files: current}, nil } - return current, nil + // Empty selector lists mean all files at runtime, never no files. + if len(legacy) == 0 { + return localSourceSelection{}, fmt.Errorf("%s", localSelectionScopeError) + } + for name, physical := range legacy { + if current[name] != physical { + return localSourceSelection{}, fmt.Errorf("%s", localSelectionScopeError) + } + } + if selection, ok := tryLocalPathSelection(inputs, legacy, current); ok { + return selection, nil + } + selection, err := proveLocalPackageSelection(root, legacy, current) + if err != nil { + return localSourceSelection{}, fmt.Errorf("proveLocalPackageSelection: %w", err) + } + return selection, nil +} + +func tryLocalPathSelection(inputs []legacyDirectory, legacy, current map[string]string) (localSourceSelection, bool) { + pathSet := make(map[string]bool) + for _, input := range inputs { + pathSet[filepath.ToSlash(filepath.Join(input.Root, input.Path))] = true + } + paths := slices.Sorted(maps.Keys(pathSet)) + // Invalid optional path syntax does not invalidate a previously supported + // package proof. Public generate.paths validation remains strict. + if v1.ValidatePathSelectors(paths) != nil { + return localSourceSelection{}, false + } + byPath := make(map[string]string) + matchedPaths := make(map[string]bool) + for name, modulePath := range current { + for _, selector := range paths { + if v1.PathSelectorMatches(selector, filepath.ToSlash(modulePath)) { + byPath[name] = modulePath + matchedPaths[selector] = true + } + } + } + if len(matchedPaths) == len(paths) && maps.Equal(legacy, byPath) { + return localSourceSelection{files: byPath, paths: paths}, true + } + return localSourceSelection{}, false +} + +func proveLocalPackageSelection(root string, legacy, current map[string]string) (localSourceSelection, error) { + packages := make(map[string]bool) + for _, name := range slices.Sorted(maps.Keys(legacy)) { + physical := legacy[name] + declared, err := readSourcePackage(root, physical) + if err != nil { + return localSourceSelection{}, fmt.Errorf("readSourcePackage: %w", err) + } + if declared == "" { + return localSourceSelection{}, fmt.Errorf("%s", localSelectionScopeError) + } + if err := v1.ValidatePackageSelectors([]string{declared}); err != nil { + return localSourceSelection{}, fmt.Errorf("ValidatePackageSelectors: %w", err) + } + packages[declared] = true + } + selected := make(map[string]string) + for _, name := range slices.Sorted(maps.Keys(current)) { + physical := current[name] + declared, err := readSourcePackage(root, physical) + if err != nil { + return localSourceSelection{}, fmt.Errorf("readSourcePackage: %w", err) + } + if packages[declared] { + selected[name] = physical + } + } + if !maps.Equal(legacy, selected) { + return localSourceSelection{}, fmt.Errorf("%s", localSelectionScopeError) + } + return localSourceSelection{files: selected, packages: slices.Sorted(maps.Keys(packages))}, nil +} + +func readSourcePackage(root, name string) (string, error) { + raw, err := sourceview.ReadLocal(context.Background(), root, name) + if err != nil { + return "", fmt.Errorf("ReadFile: %w", err) + } + return protosource.Package(raw), nil } func collectProto(root, importRoot, search string, native bool, files map[string]string) error { - // WalkDir does not follow links, but a linked starting directory/parent must - // also be rejected before traversal. rel, err := filepath.Rel(root, search) if err != nil { return fmt.Errorf("Rel: %w", err) } - part := root - for _, segment := range strings.Split(rel, string(filepath.Separator)) { - part = filepath.Join(part, segment) - info, err := os.Lstat(part) - if err != nil { - return fmt.Errorf("Lstat: %w", err) + if !filepath.IsLocal(rel) { + return fmt.Errorf("source %q leaves migration root", search) + } + rootRel, err := filepath.Rel(root, importRoot) + if err != nil { + return fmt.Errorf("Rel: %w", err) + } + rootResolution, err := sourceview.ResolveLocal(context.Background(), root, rootRel) + if err != nil { + return fmt.Errorf("ResolveLocal: %w", err) + } + physicalImportRoot := filepath.Join(root, filepath.FromSlash(rootResolution.Path)) + return sourceview.WalkLocal(context.Background(), root, rel, func(logical string, resolved sourceview.Resolution, walkErr error) error { + name := filepath.Join(root, filepath.FromSlash(logical)) + if errors.Is(walkErr, sourceview.ErrCycle) && resolved.Info != nil && resolved.Info.IsDir() { + if native && path_helpers.HiddenOrVendorSourcePath(importRoot, name) { + return nil + } + return walkErr } - if !info.IsDir() { - return fmt.Errorf("source %s is not a regular directory; manual migration required", part) + if native && path_helpers.ShouldSkipV1SourceDir(importRoot, name) { + return fs.SkipDir } - } - err = filepath.WalkDir(search, func(name string, entry fs.DirEntry, walkErr error) error { if walkErr != nil { + if filepath.Ext(name) != ".proto" && logical != filepath.ToSlash(rel) { + return nil + } return walkErr } - if entry.Type()&os.ModeSymlink != 0 { - return fmt.Errorf("source symlink %s prevents safe scope analysis", name) - } - if entry.IsDir() { - if native && path_helpers.ShouldSkipV1SourceDir(importRoot, name) { - return filepath.SkipDir + physical := filepath.Join(root, filepath.FromSlash(resolved.Path)) + if resolved.Info.IsDir() { + if native && path_helpers.ShouldSkipV1SourceDir(physicalImportRoot, physical) { + return fs.SkipDir } return nil } if filepath.Ext(name) != ".proto" { return nil } - if !entry.Type().IsRegular() { + if !resolved.Info.Mode().IsRegular() { return fmt.Errorf("source %s is not a regular file", name) } + if native { + for dir := filepath.Dir(physical); dir != root && sourceWithin(root, dir); dir = filepath.Dir(dir) { + if path_helpers.ShouldSkipV1SourceDir(physicalImportRoot, dir) { + return nil + } + } + } importName, err := filepath.Rel(importRoot, name) if err != nil { return fmt.Errorf("Rel: %w", err) } - physicalName, err := filepath.Rel(root, name) + logicalName, err := filepath.Rel(root, name) if err != nil { return fmt.Errorf("Rel: %w", err) } importName = filepath.ToSlash(importName) - if previous, ok := files[importName]; ok && previous != physicalName { - return fmt.Errorf("roots collide on import %s (%s and %s); manual migration required", importName, previous, physicalName) + if previous, ok := files[importName]; ok && previous != logicalName { + return fmt.Errorf("roots collide on import %s (%s and %s); manual migration required", importName, previous, logicalName) } - files[importName] = physicalName + files[importName] = logicalName return nil }) - if err != nil { - return fmt.Errorf("WalkDir: %w", err) - } - return nil +} + +func sourceWithin(root, name string) bool { + rel, err := filepath.Rel(root, name) + return err == nil && filepath.IsLocal(rel) } diff --git a/internal/migration/validate.go b/internal/migration/validate.go index 1d1dcd16..77653895 100644 --- a/internal/migration/validate.go +++ b/internal/migration/validate.go @@ -1,6 +1,7 @@ package migration import ( + "bytes" "fmt" "strings" @@ -47,6 +48,17 @@ func validateGenerate(raw []byte) error { if gen.Version != "" && gen.Version != "v1" { return fmt.Errorf("unsupported generation version %q", gen.Version) } + for i, module := range gen.Generate.Modules { + if err := module.Validate(); err != nil { + return fmt.Errorf("generate.modules[%d]: %w", i, err) + } + } + if err := v1.ValidatePackageSelectors(gen.Generate.Packages); err != nil { + return fmt.Errorf("ValidatePackageSelectors: %w", err) + } + if err := v1.ValidatePathSelectors(gen.Generate.Paths); err != nil { + return fmt.Errorf("ValidatePathSelectors: %w", err) + } if err := gen.Generate.Managed.Validate(); err != nil { return fmt.Errorf("Validate: %w", err) } @@ -59,12 +71,9 @@ func validateGenerate(raw []byte) error { } func validateNativeLock(raw []byte) (v1.Lock, error) { - var lock v1.Lock - if err := decodeStrict(raw, &lock); err != nil { - return v1.Lock{}, fmt.Errorf("decodeStrict: %w", err) - } - if err := lock.Validate(); err != nil { - return v1.Lock{}, fmt.Errorf("Validate: %w", err) + lock, err := v1.ParseLock(bytes.NewReader(raw)) + if err != nil { + return v1.Lock{}, fmt.Errorf("ParseLock: %w", err) } return lock, nil } diff --git a/internal/migration/validate_test.go b/internal/migration/validate_test.go new file mode 100644 index 00000000..e9efddc8 --- /dev/null +++ b/internal/migration/validate_test.go @@ -0,0 +1,175 @@ +package migration + +import ( + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + v1 "github.com/easyp-tech/easyp/internal/config/v1" +) + +func TestBuildNativeBSRLockNoOp(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name string + resolveLock bool + }{ + {name: "without_lock_resolution"}, + {name: "with_lock_resolution", resolveLock: true}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root, files, infos := nativeLockProject(t, nativeBSRLockFixture()) + repository := &mockRepository{} + + plan, err := Build(t.Context(), Options{ + Dir: root, Module: "example.com/acme/api", ResolveLock: tt.resolveLock, Repository: repository, + }) + + require.NoError(t, err) + assert.True(t, plan.AlreadyV1()) + assert.False(t, plan.NeedsLockResolution()) + assert.Empty(t, plan.Outputs()) + assert.Empty(t, repository.calls, "native validation must not access dependencies") + assertNativeLockProjectUnchanged(t, root, files, infos) + + lock, err := validateNativeLock(mustRead(t, root, v1.LockFile)) + require.NoError(t, err) + assert.Equal(t, v1.Lock{Version: 1, Modules: []v1.LockedModule{ + { + Source: "example.com/acme/dep", Version: "v1.0.0", Commit: testCommit, Hash: testHash, + BSR: []v1.BSRResolution{{ + Dependency: v1.BSRDependency{ + Module: "buf.build/googleapis/googleapis", Reference: "stable", Commit: strings.Repeat("b", 32), + Digest: "b5:" + strings.Repeat("c", 64), Config: "proto/buf.yaml", + }, + Git: v1.Requirement{Module: "github.com/googleapis/googleapis", Version: "v1.0.0"}, + Resolution: v1.BSRCompatibilitySnapshot, + }}, + }, + {Source: "github.com/googleapis/googleapis", Version: "v1.0.0", Commit: testCommit, Hash: testHash}, + }}, lock) + + require.NoError(t, plan.CheckUnchanged()) + require.NoError(t, plan.Apply()) + assert.Empty(t, repository.calls, "applying a native no-op must not refresh dependencies") + assertNativeLockProjectUnchanged(t, root, files, infos) + }) + } +} + +func TestBuildNativeLockRejectsInvalidDocuments(t *testing.T) { + t.Parallel() + valid := nativeBSRLockFixture() + for _, tt := range []struct { + name string + lock string + wantErr string + }{ + {name: "unknown_lock_field", lock: valid + "unsupported: true\n", wantErr: "field unsupported not found"}, + {name: "unknown_module_field", lock: strings.Replace(valid, " bsr:", " unsupported: true\n bsr:", 1), wantErr: "field unsupported not found"}, + {name: "unknown_binding_field", lock: strings.Replace(valid, " resolution:", " unsupported: true\n resolution:", 1), wantErr: "field unsupported not found"}, + {name: "unknown_dependency_field", lock: strings.Replace(valid, " config:", " unsupported: true\n config:", 1), wantErr: "field unsupported not found"}, + {name: "unknown_git_field", lock: strings.Replace(valid, " git: {module:", " git: {unsupported: true, module:", 1), wantErr: "field unsupported not found"}, + {name: "malformed_yaml", lock: "version: 1\nmodules: [\n", wantErr: "Decode:"}, + {name: "multiple_documents", lock: valid + "---\nversion: 1\nmodules: []\n", wantErr: "must contain one YAML document"}, + {name: "malformed_trailing_document", lock: valid + "---\nmodules: [\n", wantErr: "Decode:"}, + {name: "invalid_lock_version", lock: strings.Replace(valid, "version: 1", "version: 2", 1), wantErr: "unsupported protobuf.lock version"}, + {name: "invalid_commit", lock: strings.Replace(valid, testCommit, "invalid-commit", 1), wantErr: "invalid commit"}, + {name: "invalid_hash", lock: strings.Replace(valid, testHash, "h1:invalid", 1), wantErr: "invalid content hash"}, + {name: "unpinned_binding", lock: strings.Replace(valid, "git: {module: github.com/googleapis/googleapis, version: v1.0.0}", "git: {module: github.com/googleapis/googleapis}", 1), wantErr: "unpinned BSR Git target"}, + {name: "invalid_binding_version", lock: strings.Replace(valid, "git: {module: github.com/googleapis/googleapis, version: v1.0.0}", "git: {module: github.com/googleapis/googleapis, version: main}", 1), wantErr: "invalid version"}, + {name: "invalid_binding_resolution", lock: strings.Replace(valid, "compatibility_snapshot", "implicit_latest", 1), wantErr: "invalid BSR resolution"}, + {name: "invalid_binding_origin", lock: strings.Replace(valid, "config: proto/buf.yaml", "config: ../buf.yaml", 1), wantErr: "invalid BSR config path"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + root, files, infos := nativeLockProject(t, tt.lock) + repository := &mockRepository{} + + plan, err := Build(t.Context(), Options{ + Dir: root, Module: "example.com/acme/api", ResolveLock: true, Repository: repository, + }) + + require.ErrorContains(t, err, "checkNative: validateNativeLock:") + require.ErrorContains(t, err, tt.wantErr) + assert.Nil(t, plan) + assert.Empty(t, repository.calls, "invalid native locks must not access dependencies") + assertNativeLockProjectUnchanged(t, root, files, infos) + }) + } +} + +func nativeBSRLockFixture() string { + return fmt.Sprintf(`version: 1 +modules: + - source: example.com/acme/dep + version: v1.0.0 + commit: %s + hash: %s + bsr: + - dependency: + module: buf.build/googleapis/googleapis + reference: stable + commit: %s + digest: b5:%s + config: proto/buf.yaml + git: {module: github.com/googleapis/googleapis, version: v1.0.0} + resolution: compatibility_snapshot + - source: github.com/googleapis/googleapis + version: v1.0.0 + commit: %s + hash: %s +`, testCommit, testHash, strings.Repeat("b", 32), strings.Repeat("c", 64), testCommit, testHash) +} + +func nativeLockProject(t *testing.T, lock string) (string, map[string]string, map[string]os.FileInfo) { + t.Helper() + root := t.TempDir() + files := map[string]string{ + v1.PolicyFile: "# Keep native policy bytes.\nversion: v1\n", + v1.GenerateFile: "# Keep native generation bytes.\nversion: v1\nplugins: []\n", + v1.ModuleFile: "// Keep native manifest bytes.\nmodule example.com/acme/api\nrequire example.com/acme/dep v1.0.0\n", + v1.LockFile: lock, + "api.proto": "syntax = \"proto3\";\npackage acme.api;\nmessage Request {}\n", + "README.md": "Unrelated project content must also stay unchanged.\n", + } + infos := make(map[string]os.FileInfo, len(files)+1) + for name, content := range files { + writeFixture(t, root, name, content) + info, err := os.Stat(filepath.Join(root, name)) + require.NoError(t, err) + infos[name] = info + } + info, err := os.Stat(root) + require.NoError(t, err) + infos["."] = info + return root, files, infos +} + +func assertNativeLockProjectUnchanged(t *testing.T, root string, files map[string]string, infos map[string]os.FileInfo) { + t.Helper() + entries, err := os.ReadDir(root) + require.NoError(t, err) + var actual, expected []string + for _, entry := range entries { + actual = append(actual, entry.Name()) + } + for name, content := range files { + expected = append(expected, name) + assert.Equal(t, content, string(mustRead(t, root, name)), name) + } + assert.ElementsMatch(t, expected, actual, "native migration must not create backups or staging files") + for name, before := range infos { + after, err := os.Stat(filepath.Join(root, name)) + require.NoError(t, err) + assert.True(t, os.SameFile(before, after), "%s was replaced", name) + assert.Equal(t, before.Mode(), after.Mode(), "%s mode changed", name) + assert.Equal(t, before.ModTime(), after.ModTime(), "%s was written", name) + } +} diff --git a/internal/modules/aliases_test.go b/internal/modules/aliases_test.go new file mode 100644 index 00000000..cd29105b --- /dev/null +++ b/internal/modules/aliases_test.go @@ -0,0 +1,108 @@ +package modules + +import ( + "os" + "path/filepath" + "testing" + + v1 "github.com/easyp-tech/easyp/internal/config/v1" + "github.com/easyp-tech/easyp/internal/sourceview" + "github.com/stretchr/testify/require" +) + +func TestSourceRootWalkPreservesInternalDirectoryAlias(t *testing.T) { + t.Parallel() + directory := t.TempDir() + writeV1GenerateFixture(t, directory, "sources/api/model.proto", "syntax = \"proto3\";\n") + require.NoError(t, os.Symlink("sources", filepath.Join(directory, "proto"))) + roots, err := ModuleSources(directory, v1.Module{Name: "example.com/api", Roots: []string{"proto"}}) + require.NoError(t, err) + var paths []string + require.NoError(t, roots[0].Walk(func(path string) error { paths = append(paths, path); return nil })) + require.Equal(t, []string{filepath.Join(directory, "proto/api/model.proto")}, paths) +} + +func TestSourceRootWalkRejectsExternalProtoAlias(t *testing.T) { + t.Parallel() + directory, outside := t.TempDir(), t.TempDir() + writeV1GenerateFixture(t, outside, "secret.proto", "syntax = \"proto3\";\n") + require.NoError(t, os.Symlink(filepath.Join(outside, "secret.proto"), filepath.Join(directory, "alias.proto"))) + roots, err := ModuleSources(directory, v1.Module{Name: "example.com/api", Roots: []string{"."}}) + require.NoError(t, err) + require.Error(t, roots[0].Walk(func(string) error { return nil })) +} + +func TestFilteredFileDoesNotSkipSelectedSiblings(t *testing.T) { + t.Parallel() + directory := t.TempDir() + writeV1GenerateFixture(t, directory, "a.proto", "syntax = \"proto3\";\n") + writeV1GenerateFixture(t, directory, "b.proto", "syntax = \"proto3\";\n") + roots, err := ModuleSources(directory, v1.Module{Name: "example.com/api", Roots: []string{"."}, ProtoFilters: []v1.ProtoFileFilter{{Root: ".", Excludes: []string{"a.proto"}}}}) + require.NoError(t, err) + var paths []string + require.NoError(t, roots[0].Walk(func(path string) error { paths = append(paths, path); return nil })) + require.Equal(t, []string{filepath.Join(directory, "b.proto")}, paths) +} + +func TestDeclaredDependencyAliasAcrossNestedRepository(t *testing.T) { + t.Parallel() + root := t.TempDir() + writeV1GenerateFixture(t, root, "protobuf.mod", "module example.test/app\nroots .\nrequire example.test/dep\nreplace example.test/dep => ./dep\n") + writeV1GenerateFixture(t, root, "dep/.git", "gitdir: /unused\n") + writeV1GenerateFixture(t, root, "dep/buf.yaml", "version: v2\nmodules:\n - path: .\n includes: [selected]\n") + writeV1GenerateFixture(t, root, "dep/selected/model.proto", "syntax = \"proto3\";\n") + writeV1GenerateFixture(t, root, "client.proto", "syntax = \"proto3\"; import \"alias.proto\";\n") + require.NoError(t, os.Symlink("dep/selected/model.proto", filepath.Join(root, "alias.proto"))) + require.NoError(t, Tidy(t.Context(), root, nil)) +} + +func TestImportRootAliasPreservesPhysicalBufExclusions(t *testing.T) { + t.Parallel() + directory := t.TempDir() + writeV1GenerateFixture(t, directory, "real/allowed/keep.proto", "syntax = \"proto3\";\n") + writeV1GenerateFixture(t, directory, "real/excluded/blocked.proto", "syntax = \"proto3\";\n") + require.NoError(t, os.Symlink("real", filepath.Join(directory, "proto"))) + require.NoError(t, os.Symlink("../excluded/blocked.proto", filepath.Join(directory, "real/allowed/alias.proto"))) + sources, err := ModuleSources(directory, v1.Module{Name: "example.com/api", Roots: []string{"proto"}, ProtoFilters: []v1.ProtoFileFilter{{Root: "proto", Excludes: []string{"proto/excluded"}}}}) + require.NoError(t, err) + require.False(t, sources.FileAllowed()(filepath.Join(directory, "proto/allowed/alias.proto"))) + owners, err := sources.FileModules() + require.NoError(t, err) + require.Equal(t, map[string]string{"allowed/keep.proto": "example.com/api"}, owners) +} + +func TestSourceRootDirectoryAliasCycleFails(t *testing.T) { + t.Parallel() + directory := t.TempDir() + writeV1GenerateFixture(t, directory, "file.proto", "syntax = \"proto3\";\n") + writeV1GenerateFixture(t, directory, "protobuf.mod", "module example.com/api\nroots proto\n") + require.NoError(t, os.Symlink(".", filepath.Join(directory, "proto"))) + sources, err := ModuleSources(directory, v1.Module{Name: "example.com/api", Roots: []string{"proto"}}) + require.NoError(t, err) + require.ErrorIs(t, sources[0].Walk(func(string) error { return nil }), sourceview.ErrCycle) +} + +func TestGraphAliasCannotReadUndeclaredNestedRepository(t *testing.T) { + t.Parallel() + directory := t.TempDir() + writeV1GenerateFixture(t, directory, "dep/.git", "gitdir: /unused\n") + writeV1GenerateFixture(t, directory, "dep/file.proto", "syntax = \"proto3\";\n") + require.NoError(t, os.Mkdir(filepath.Join(directory, "proto"), 0o755)) + require.NoError(t, os.Symlink("../dep/file.proto", filepath.Join(directory, "proto/alias.proto"))) + sources, err := ModuleSources(directory, v1.Module{Name: "example.com/api", Roots: []string{"proto"}}) + require.NoError(t, err) + require.ErrorIs(t, sources.WalkSelected(sources[0], nil, func(string) error { return nil }), sourceview.ErrNestedRepository) + _, err = sources.ReadSourceFile(filepath.Join(directory, "proto/alias.proto")) + require.ErrorIs(t, err, sourceview.ErrNestedRepository) +} + +func TestFilteredDirectoryAliasCycleRemainsStrict(t *testing.T) { + t.Parallel() + directory := t.TempDir() + writeV1GenerateFixture(t, directory, "allowed/file.proto", "syntax = \"proto3\";\n") + require.NoError(t, os.Symlink(".", filepath.Join(directory, "cycle"))) + sources, err := ModuleSources(directory, v1.Module{Name: "example.com/api", Roots: []string{"."}, ProtoFilters: []v1.ProtoFileFilter{{Root: ".", Includes: []string{"allowed", "cycle/allowed"}}}}) + require.NoError(t, err) + _, err = sources.FileModules() + require.ErrorIs(t, err, sourceview.ErrCycle) +} diff --git a/internal/modules/collisions.go b/internal/modules/collisions.go index e493f3ad..2d389081 100644 --- a/internal/modules/collisions.go +++ b/internal/modules/collisions.go @@ -10,7 +10,7 @@ func CheckSourceCollisions(roots SourceRoots) error { seen := make(map[string]string) allowed := roots.FileAllowed() for _, root := range roots { - err := root.Walk(func(path string) error { + err := root.walk(allowed, true, roots.ownsSelectedPhysical, func(path string) error { if allowed != nil && !allowed(path) { return nil } diff --git a/internal/modules/imports.go b/internal/modules/imports.go index fbe8b1a0..472715a0 100644 --- a/internal/modules/imports.go +++ b/internal/modules/imports.go @@ -2,6 +2,7 @@ package modules import ( "bytes" + "context" "errors" "fmt" "io/fs" @@ -14,12 +15,22 @@ import ( "github.com/bufbuild/protocompile/parser" "github.com/bufbuild/protocompile/reporter" + "github.com/easyp-tech/easyp/internal/sourceview" + "github.com/easyp-tech/easyp/internal/workspace" "github.com/easyp-tech/easyp/wellknownimports" ) // ReadProtoImports reads and parses import declarations in a proto file. func ReadProtoImports(path string) ([]string, error) { - raw, err := os.ReadFile(path) + boundary, err := workspace.Boundary(filepath.Dir(path)) + if err != nil { + return nil, fmt.Errorf("Boundary: %w", err) + } + relative, err := filepath.Rel(boundary, path) + if err != nil { + return nil, fmt.Errorf("Rel: %w", err) + } + raw, err := sourceview.ReadLocal(context.Background(), boundary, relative) if err != nil { return nil, fmt.Errorf("ReadFile: %w", err) } @@ -82,9 +93,13 @@ type v1ImportSource struct { builtin bool } -func (s v1ImportSource) imports() ([]string, error) { +func (s v1ImportSource) imports(roots SourceRoots) ([]string, error) { if !s.builtin { - return ReadProtoImports(s.path) + raw, err := roots.ReadSourceFile(s.path) + if err != nil { + return nil, fmt.Errorf("ReadSourceFile: %w", err) + } + return ParseProtoImports(s.path, raw) } raw, err := wellknownimports.Content.ReadFile(s.path) if err != nil { @@ -106,14 +121,14 @@ func findUnresolvedV1Imports(moduleDir string, roots, dependencyRoots []string) func findUnresolvedV1ImportsWithSources(moduleDir string, roots []string, dependencyRoots SourceRoots) ([]v1UnresolvedImport, error) { allRoots := make(SourceRoots, 0, len(roots)+len(dependencyRoots)) for _, root := range roots { - allRoots = append(allRoots, SourceRoot{Path: filepath.Join(moduleDir, root)}) + allRoots = append(allRoots, SourceRoot{Path: filepath.Join(moduleDir, root), directory: moduleDir}) } allRoots = append(allRoots, dependencyRoots...) allowed := allRoots.FileAllowed() var queue []v1ImportSource seen := make(map[v1ImportSource]bool) for _, sourceRoot := range allRoots[:len(roots)] { - err := sourceRoot.Walk(func(path string) error { + err := allRoots.WalkSelected(sourceRoot, allowed, func(path string) error { if allowed != nil && !allowed(path) { return nil } @@ -132,7 +147,7 @@ func findUnresolvedV1ImportsWithSources(moduleDir string, roots []string, depend for len(queue) > 0 { source := queue[0] queue = queue[1:] - imports, err := source.imports() + imports, err := source.imports(allRoots) if err != nil { return nil, fmt.Errorf("imports: %w", err) } @@ -174,7 +189,15 @@ func resolveV1ImportSource(importPath string, roots SourceRoots, allowed func(st } else if !root.allows(candidate, root.Path) { continue } - info, err := os.Stat(candidate) + relative, err := filepath.Rel(root.boundary(), candidate) + if err != nil { + return v1ImportSource{}, fmt.Errorf("Rel: %w", err) + } + resolved, err := sourceview.ResolveLocal(context.Background(), root.boundary(), relative) + info := resolved.Info + if errors.Is(err, sourceview.ErrNestedRepository) && roots.ownsSelectedPhysical(filepath.Join(physicalSourcePath(root.boundary()), filepath.FromSlash(resolved.Path))) { + err = nil + } if errors.Is(err, os.ErrNotExist) { continue } diff --git a/internal/modules/local_sources.go b/internal/modules/local_sources.go index 892449b2..d5724a4a 100644 --- a/internal/modules/local_sources.go +++ b/internal/modules/local_sources.go @@ -2,6 +2,8 @@ package modules import ( "bytes" + "context" + "errors" "fmt" "os" "path/filepath" @@ -9,6 +11,7 @@ import ( moduleconfig "github.com/easyp-tech/easyp/internal/adapters/module_config" v1 "github.com/easyp-tech/easyp/internal/config/v1" + "github.com/easyp-tech/easyp/internal/sourceview" ) type localReplacements struct { @@ -77,8 +80,8 @@ func readLocalReplacement(directory, name string) (EffectiveModule, error) { if !info.IsDir() { return EffectiveModule{}, fmt.Errorf("%s is not a directory", directory) } - raw, err := os.ReadFile(filepath.Join(directory, v1.ModuleFile)) - if err != nil && !os.IsNotExist(err) { + raw, err := sourceview.ReadLocal(context.Background(), directory, v1.ModuleFile) + if err != nil && !errors.Is(err, os.ErrNotExist) { return EffectiveModule{}, fmt.Errorf("ReadFile: %w", err) } var module v1.Module @@ -92,7 +95,7 @@ func readLocalReplacement(directory, name string) (EffectiveModule, error) { } } else { // Compatibility reads only roots/requirements, never dependency plugins. - module, err = moduleconfig.ReadGitDependency(directory, name) + module, err = moduleconfig.ReadLocalDependency(directory, name) if err != nil { return EffectiveModule{}, fmt.Errorf("ReadGitDependency: %w", err) } diff --git a/internal/modules/locked_sources.go b/internal/modules/locked_sources.go index 41ff81ea..1115cb6f 100644 --- a/internal/modules/locked_sources.go +++ b/internal/modules/locked_sources.go @@ -3,6 +3,7 @@ package modules import ( "bytes" "context" + "errors" "fmt" "os" "path/filepath" @@ -11,6 +12,7 @@ import ( "golang.org/x/mod/semver" v1 "github.com/easyp-tech/easyp/internal/config/v1" + "github.com/easyp-tech/easyp/internal/sourceview" ) // EnsureLockedSources loads import roots from verified cached modules @@ -65,8 +67,8 @@ func unreplacedRequirements(requirements []v1.Requirement, replaced map[string]b // ReadLock parses a lockfile without installing or changing dependencies. func ReadLock(path string) (v1.Lock, error) { - raw, err := os.ReadFile(path) - if os.IsNotExist(err) { + raw, err := sourceview.ReadLocal(context.Background(), filepath.Dir(path), filepath.Base(path)) + if errors.Is(err, os.ErrNotExist) { if _, legacyErr := os.Stat(filepath.Join(filepath.Dir(path), "easyp.lock")); legacyErr == nil { return v1.Lock{}, fmt.Errorf("%w: easyp.lock needs verified conversion, not renaming", v1.ErrLegacyConfiguration) } diff --git a/internal/modules/manifest_requirements.go b/internal/modules/manifest_requirements.go index 8fb397d8..41b95d42 100644 --- a/internal/modules/manifest_requirements.go +++ b/internal/modules/manifest_requirements.go @@ -101,9 +101,13 @@ func augmentV1ManifestRequirements(original []byte, root string, module v1.Modul func v1RootImports(moduleDir string, roots []string) (map[string]bool, error) { imports := make(map[string]bool) for _, root := range roots { - sourceRoot := filepath.Join(moduleDir, root) - err := WalkProtoFiles(sourceRoot, func(path string) error { - paths, err := ReadProtoImports(path) + sourceRoot := SourceRoot{Path: filepath.Join(moduleDir, root), directory: moduleDir} + err := sourceRoot.Walk(func(path string) error { + raw, err := (SourceRoots{sourceRoot}).ReadSourceFile(path) + if err != nil { + return fmt.Errorf("ReadSourceFile: %w", err) + } + paths, err := ParseProtoImports(path, raw) if err != nil { return fmt.Errorf("ReadProtoImports: %w", err) } diff --git a/internal/modules/project_files.go b/internal/modules/project_files.go index 72faaee3..673de738 100644 --- a/internal/modules/project_files.go +++ b/internal/modules/project_files.go @@ -2,6 +2,7 @@ package modules import ( "bytes" + "context" "errors" "fmt" "os" @@ -11,11 +12,12 @@ import ( v1 "github.com/easyp-tech/easyp/internal/config/v1" disk "github.com/easyp-tech/easyp/internal/fs/fs" + "github.com/easyp-tech/easyp/internal/sourceview" ) // ReadManifest returns parsed metadata and the original bytes for comment-preserving edits. func ReadManifest(root string) ([]byte, v1.Module, error) { - original, err := os.ReadFile(filepath.Join(root, v1.ModuleFile)) + original, err := sourceview.ReadLocal(context.Background(), root, v1.ModuleFile) if err != nil { return nil, v1.Module{}, fmt.Errorf("ReadFile: %w", err) } diff --git a/internal/modules/sources.go b/internal/modules/sources.go index 58f2ff59..5e1f2c6c 100644 --- a/internal/modules/sources.go +++ b/internal/modules/sources.go @@ -1,7 +1,11 @@ package modules import ( + "context" + "errors" "fmt" + "io" + "io/fs" "os" "path/filepath" "slices" @@ -9,13 +13,16 @@ import ( v1 "github.com/easyp-tech/easyp/internal/config/v1" "github.com/easyp-tech/easyp/internal/core/path_helpers" + "github.com/easyp-tech/easyp/internal/sourceview" ) // SourceRoot associates a physical import root with its module identity. type SourceRoot struct { - Path string - Module string - fileAllowed func(string) bool + Path string + Module string + fileAllowed func(string) bool + directoryAllowed func(string) bool + directory string } // SourceRoots preserves import precedence and module ownership. @@ -31,15 +38,124 @@ func (roots SourceRoots) Paths() []string { } // Walk visits only proto files selected by the module's source metadata. -func (root SourceRoot) Walk(visit func(string) error) error { - return WalkProtoFiles(root.Path, func(path string) error { - if root.fileAllowed != nil && (!root.fileAllowed(path) || !root.fileAllowed(physicalSourcePath(path))) { +func (root SourceRoot) Walk(visit func(string) error) error { return root.WalkSelected(nil, visit) } + +// WalkSelected applies logical selection before reading or reporting file alias errors. +func (root SourceRoot) WalkSelected(selected func(string) bool, visit func(string) error) error { + return root.walk(selected, false, nil, visit) +} + +// WalkSelected walks one root with the graph's logical and target ownership. +func (roots SourceRoots) WalkSelected(root SourceRoot, selected func(string) bool, visit func(string) error) error { + return root.walk(selected, false, roots.ownsSelectedPhysical, visit) +} + +func (root SourceRoot) walk(selected func(string) bool, namesOnly bool, targetAllowed func(string) bool, visit func(string) error) error { + boundary := root.boundary() + relative, err := filepath.Rel(boundary, root.Path) + if err != nil { + return fmt.Errorf("Rel: %w", err) + } + physicalRoot, err := sourceview.ResolveLocal(context.Background(), boundary, relative) + if err != nil { + return fmt.Errorf("ResolveLocal: %w", err) + } + return sourceview.WalkLocal(context.Background(), boundary, relative, func(logical string, resolved sourceview.Resolution, walkErr error) error { + path := filepath.Join(boundary, filepath.FromSlash(logical)) + if filepath.Ext(path) == ".proto" && root.fileAllowed != nil && !root.fileAllowed(path) { + return nil + } + if errors.Is(walkErr, sourceview.ErrCycle) && resolved.Info != nil && resolved.Info.IsDir() { + if path_helpers.HiddenOrVendorSourcePath(root.Path, path) || (root.directoryAllowed != nil && !root.directoryAllowed(path)) { + return nil + } + return walkErr + } + if root.directoryAllowed != nil && resolved.Info != nil && resolved.Info.IsDir() && !root.directoryAllowed(path) { + return fs.SkipDir + } + if path_helpers.ShouldSkipV1SourceDir(root.Path, path) { + return fs.SkipDir + } + if filepath.Ext(path) == ".proto" && selected != nil && !selected(path) { + return nil + } + if errors.Is(walkErr, sourceview.ErrNestedRepository) && targetAllowed != nil && targetAllowed(filepath.Join(physicalSourcePath(boundary), filepath.FromSlash(resolved.Path))) { + walkErr = nil + } + if walkErr != nil && namesOnly && filepath.Ext(path) == ".proto" { + return visit(path) + } + if walkErr != nil { + if filepath.Ext(path) != ".proto" && path != root.Path { + return nil + } + return walkErr + } + physical := filepath.Join(physicalSourcePath(boundary), filepath.FromSlash(resolved.Path)) + if !resolved.Info.IsDir() && root.fileAllowed != nil && !root.fileAllowed(physical) { + return nil + } + if resolved.Info.IsDir() { + if path_helpers.ShouldSkipV1SourceDir(filepath.Join(physicalSourcePath(boundary), filepath.FromSlash(physicalRoot.Path)), physical) { + return fs.SkipDir + } + return nil + } + if filepath.Ext(path) != ".proto" { return nil } return visit(path) }) } +func (root SourceRoot) boundary() string { + if root.directory != "" { + return root.directory + } + return root.Path +} + +// OpenSourceFile opens a source at its logical location through its owning root. +func (roots SourceRoots) OpenSourceFile(path string) (io.ReadCloser, error) { + allowed := roots.FileAllowed() + if allowed != nil && !allowed(path) { + return nil, &os.PathError{Op: "open", Path: path, Err: os.ErrNotExist} + } + for _, root := range roots { + if !sourcePathWithin(path, root.Path) { + continue + } + relative, err := filepath.Rel(root.boundary(), path) + if err != nil { + return nil, fmt.Errorf("Rel: %w", err) + } + physicalPath := func(resolved sourceview.Resolution) string { + return filepath.Join(physicalSourcePath(root.boundary()), filepath.FromSlash(resolved.Path)) + } + file, err := sourceview.OpenLocalSelected(context.Background(), root.boundary(), relative, func(resolved sourceview.Resolution) bool { return allowed == nil || allowed(physicalPath(resolved)) }, func(resolved sourceview.Resolution) bool { return roots.ownsSelectedPhysical(physicalPath(resolved)) }) + if err != nil { + return nil, fmt.Errorf("OpenLocal: %w", err) + } + return file, nil + } + return nil, &os.PathError{Op: "open", Path: path, Err: os.ErrNotExist} +} + +// ReadSourceFile reads a selected logical source through its owning root. +func (roots SourceRoots) ReadSourceFile(path string) (_ []byte, resultErr error) { + file, err := roots.OpenSourceFile(path) + if err != nil { + return nil, fmt.Errorf("OpenSourceFile: %w", err) + } + defer func() { resultErr = errors.Join(resultErr, file.Close()) }() + data, err := io.ReadAll(file) + if err != nil { + return nil, fmt.Errorf("ReadAll: %w", err) + } + return data, nil +} + func (root SourceRoot) allows(path, importRoot string) bool { return root.allowsPath(filepath.Clean(path), filepath.Clean(importRoot)) && root.allowsPath(physicalSourcePath(path), physicalSourcePath(importRoot)) @@ -64,9 +180,7 @@ func (root SourceRoot) allowsPath(path, importRoot string) bool { // Overlapping roots select the union of their allowed files. Both the import's // spelling and its physical target must pass their respective owning selections. func (roots SourceRoots) FileAllowed() func(string) bool { - if !slices.ContainsFunc(roots, func(root SourceRoot) bool { return root.fileAllowed != nil }) { - return nil - } + type selection struct { path string root SourceRoot @@ -110,7 +224,7 @@ func physicalSourcePath(path string) string { return filepath.Clean(path) } -func moduleFileSelection(directory string, filters []v1.ProtoFileFilter) func(string) bool { +func moduleFileSelection(directory string, filters []v1.ProtoFileFilter, includeAncestors bool) func(string) bool { canonical, err := filepath.EvalSymlinks(directory) if err != nil { canonical = directory @@ -121,6 +235,30 @@ func moduleFileSelection(directory string, filters []v1.ProtoFileFilter) func(st filters[i].Includes = slices.Clone(filter.Includes) filters[i].Excludes = slices.Clone(filter.Excludes) } + // Root aliases preserve relative Buf selection in their physical namespace. + // Selection leaves stay literal, so excluding an alias does not exclude its target. + logicalFilters := slices.Clone(filters) + for _, filter := range logicalFilters { + resolved, err := sourceview.ResolveLocal(context.Background(), directory, filter.Root) + if err != nil || len(resolved.Links) == 0 { + continue + } + physical := filter + physical.Root = filepath.FromSlash(resolved.Path) + translate := func(names []string) []string { + translated := make([]string, 0, len(names)) + for _, name := range names { + relative, err := filepath.Rel(filter.Root, name) + if err != nil || !filepath.IsLocal(relative) { + continue + } + translated = append(translated, filepath.Join(physical.Root, relative)) + } + return translated + } + physical.Includes, physical.Excludes = translate(filter.Includes), translate(filter.Excludes) + filters = append(filters, physical) + } return func(path string) bool { base := directory if !sourcePathWithin(path, base) { @@ -137,11 +275,14 @@ func moduleFileSelection(directory string, filters []v1.ProtoFileFilter) func(st return directory == "." || name == directory || strings.HasPrefix(name, directory+"/") } for _, filter := range filters { - if !within(filter.Root) { + inScope := within(filter.Root) || (includeAncestors && v1.PathSelectorMatches(name, filepath.ToSlash(filter.Root))) + if !inScope { continue } inside = true - included := len(filter.Includes) == 0 || slices.ContainsFunc(filter.Includes, within) + included := len(filter.Includes) == 0 || slices.ContainsFunc(filter.Includes, func(included string) bool { + return within(included) || (includeAncestors && v1.PathSelectorMatches(name, filepath.ToSlash(included))) + }) if included && !slices.ContainsFunc(filter.Excludes, within) { return true } @@ -156,7 +297,7 @@ func (roots SourceRoots) FileModules() (map[string]string, error) { modules := make(map[string]string) allowed := roots.FileAllowed() for _, root := range roots { - err := root.Walk(func(path string) error { + err := root.walk(allowed, true, roots.ownsSelectedPhysical, func(path string) error { if allowed != nil && !allowed(path) { return nil } @@ -177,20 +318,36 @@ func (roots SourceRoots) FileModules() (map[string]string, error) { // ModuleSources validates module roots and resolves them relative to its directory. func ModuleSources(directory string, module v1.Module) (SourceRoots, error) { roots := make(SourceRoots, 0, len(module.Roots)) - var allowed func(string) bool + var allowed, dirAllowed func(string) bool if len(module.ProtoFilters) > 0 { - allowed = moduleFileSelection(directory, module.ProtoFilters) + allowed = moduleFileSelection(directory, module.ProtoFilters, false) + dirAllowed = moduleFileSelection(directory, module.ProtoFilters, true) } for _, root := range module.Roots { path := filepath.Join(directory, root) - info, err := os.Stat(path) + resolved, err := sourceview.ResolveLocal(context.Background(), directory, root) + info := resolved.Info if err != nil { return nil, fmt.Errorf("Stat: %w", err) } if !info.IsDir() { return nil, fmt.Errorf("module %s has invalid root %q: not a directory", module.Name, root) } - roots = append(roots, SourceRoot{Path: path, Module: module.Name, fileAllowed: allowed}) + boundary := directory + if filepath.Clean(path) == filepath.Clean(directory) { + boundary = "" + } + roots = append(roots, SourceRoot{Path: path, Module: module.Name, fileAllowed: allowed, directoryAllowed: dirAllowed, directory: boundary}) } return roots, nil } + +func (roots SourceRoots) ownsSelectedPhysical(path string) bool { + for _, root := range roots { + physicalRoot := physicalSourcePath(root.Path) + if sourcePathWithin(path, physicalRoot) && root.allowsPath(path, physicalRoot) { + return true + } + } + return false +} diff --git a/internal/modules/vendor.go b/internal/modules/vendor.go index bd6f9992..9df80d87 100644 --- a/internal/modules/vendor.go +++ b/internal/modules/vendor.go @@ -7,7 +7,7 @@ import ( "os" "path/filepath" - disk "github.com/easyp-tech/easyp/internal/fs/fs" + "io" ) // Vendor copies the effective dependency sources into one import root. @@ -50,8 +50,8 @@ func writeV1Vendor(root string, roots SourceRoots) error { if !filepath.IsLocal(importPath) { return fmt.Errorf("invalid import path %q", importPath) } - if err := disk.CopyRegularFile(path, filepath.Join(stage, importPath)); err != nil { - return fmt.Errorf("CopyRegularFile: %w", err) + if err := copySourceFile(roots, path, filepath.Join(stage, importPath)); err != nil { + return fmt.Errorf("copySourceFile: %w", err) } return nil }) @@ -108,3 +108,36 @@ func restoreV1Vendor(backup, target string) error { } return nil } + +func copySourceFile(roots SourceRoots, source, destination string) (resultErr error) { + in, err := roots.OpenSourceFile(source) + if err != nil { + return fmt.Errorf("OpenSourceFile: %w", err) + } + defer func() { resultErr = errors.Join(resultErr, in.Close()) }() + statter, ok := in.(interface{ Stat() (os.FileInfo, error) }) + if !ok { + return fmt.Errorf("source %q has no file metadata", source) + } + info, err := statter.Stat() + if err != nil { + return fmt.Errorf("Stat: %w", err) + } + if !info.Mode().IsRegular() { + return fmt.Errorf("source %q is not regular", source) + } + err = os.MkdirAll(filepath.Dir(destination), 0o755) + if err != nil { + return fmt.Errorf("MkdirAll: %w", err) + } + out, err := os.OpenFile(destination, os.O_CREATE|os.O_EXCL|os.O_WRONLY, info.Mode().Perm()) + if err != nil { + return fmt.Errorf("OpenFile: %w", err) + } + defer func() { resultErr = errors.Join(resultErr, out.Close()) }() + _, err = io.Copy(out, in) + if err != nil { + return fmt.Errorf("Copy: %w", err) + } + return nil +} diff --git a/internal/modules/walk.go b/internal/modules/walk.go index f3a2d3ce..40984fdb 100644 --- a/internal/modules/walk.go +++ b/internal/modules/walk.go @@ -1,27 +1,6 @@ package modules -import ( - "io/fs" - "path/filepath" - - "github.com/easyp-tech/easyp/internal/core/path_helpers" -) - // WalkProtoFiles visits source files while excluding hidden, vendor and nested-module directories. func WalkProtoFiles(root string, visit func(string) error) error { - return filepath.WalkDir(root, func(path string, entry fs.DirEntry, walkErr error) error { - if walkErr != nil { - return walkErr - } - if entry.IsDir() { - if path_helpers.ShouldSkipV1SourceDir(root, path) { - return filepath.SkipDir - } - return nil - } - if filepath.Ext(path) != ".proto" { - return nil - } - return visit(path) - }) + return (SourceRoot{Path: root}).Walk(visit) } diff --git a/internal/policy/aliases_test.go b/internal/policy/aliases_test.go new file mode 100644 index 00000000..22dc2c6f --- /dev/null +++ b/internal/policy/aliases_test.go @@ -0,0 +1,24 @@ +package policy + +import ( + "os" + "path/filepath" + "testing" + + v1 "github.com/easyp-tech/easyp/internal/config/v1" + "github.com/stretchr/testify/require" +) + +func TestLinkedPolicyUsesLogicalRelativeExtends(t *testing.T) { + t.Parallel() + root := t.TempDir() + require.NoError(t, os.MkdirAll(filepath.Join(root, "storage"), 0o755)) + require.NoError(t, os.MkdirAll(filepath.Join(root, "consumer"), 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(root, "storage/source.yaml"), []byte("version: v1\nbreaking:\n extends: ./base.yaml\n"), 0o644)) + require.NoError(t, os.WriteFile(filepath.Join(root, "consumer/base.yaml"), []byte("version: v1\nbreaking:\n baseline: git:logical\n"), 0o644)) + require.NoError(t, os.Symlink("../storage/source.yaml", filepath.Join(root, "consumer/easyp.yaml"))) + resolver := NewResolver(root, nil) + result, err := resolver.ResolveBreaking(t.Context(), BreakingInput{PolicyPath: filepath.Join(root, "consumer/easyp.yaml"), Policy: v1.Policy{Version: "v1", Breaking: v1.BreakingPolicy{Extends: "./base.yaml"}}}) + require.NoError(t, err) + require.Equal(t, "git:logical", result.Policy.Breaking.Baseline) +} diff --git a/internal/policy/resolver.go b/internal/policy/resolver.go index ec359423..cb105426 100644 --- a/internal/policy/resolver.go +++ b/internal/policy/resolver.go @@ -13,6 +13,7 @@ import ( v1 "github.com/easyp-tech/easyp/internal/config/v1" "github.com/easyp-tech/easyp/internal/modules" + "github.com/easyp-tech/easyp/internal/sourceview" ) const maxExtendsDepth = 16 @@ -34,9 +35,6 @@ type Resolver struct { // NewResolver constructs an operation-scoped policy resolver. func NewResolver(workspaceRoot string, graph GraphProvider) *Resolver { - if canonical, err := filepath.EvalSymlinks(workspaceRoot); err == nil { - workspaceRoot = canonical - } return &Resolver{ WorkspaceRoot: workspaceRoot, Graph: graph, @@ -106,7 +104,7 @@ func (r *Resolver) ResolveLint(ctx context.Context, input LintInput) (LintResult if err != nil { return LintResult{}, fmt.Errorf("resolve %s linters.extends %q for consuming policy %s: %w", input.PolicyPath, local.Linters.Extends, input.PolicyPath, err) } - current, err := r.load(input.PolicyPath, true) + current, err := r.load(input.PolicyPath, r.WorkspaceRoot, true) if err != nil { return LintResult{}, err } @@ -150,7 +148,7 @@ func (r *Resolver) ResolveBreaking(ctx context.Context, input BreakingInput) (Br if err != nil { return BreakingResult{}, fmt.Errorf("resolve %s breaking.extends %q for consuming policy %s: %w", input.PolicyPath, local.Breaking.Extends, input.PolicyPath, err) } - current, err := r.load(input.PolicyPath, true) + current, err := r.load(input.PolicyPath, r.WorkspaceRoot, true) if err != nil { return BreakingResult{}, err } @@ -168,11 +166,11 @@ func (r *Resolver) ResolveBreaking(ctx context.Context, input BreakingInput) (Br } func (r *Resolver) resolveLintFile(ctx context.Context, path, boundary string, graphKey string, expandEnvironment bool, stack []string) (resolvedSection, error) { - loaded, err := r.load(path, expandEnvironment) + loaded, err := r.load(path, boundary, expandEnvironment) if err != nil { return resolvedSection{}, err } - key := resolveKey{path: loaded.canonical, section: v1.PolicySectionLinters, graph: graphKey, boundary: boundary, expandEnvironment: loaded.expandEnvironment} + key := resolveKey{path: loaded.path, section: v1.PolicySectionLinters, graph: graphKey, boundary: boundary, expandEnvironment: loaded.expandEnvironment} if cached, ok := r.resolved[key]; ok { if len(stack)+len(cached.chain) > maxExtendsDepth { return resolvedSection{}, fmt.Errorf("policy extends exceeds maximum depth %d at %s", maxExtendsDepth, path) @@ -202,22 +200,22 @@ func (r *Resolver) resolveLintFile(ctx context.Context, path, boundary string, g } result.linters = v1.MergeLinterSection(base.linters, loaded.policy.Linters, loaded.presence, true) result.settings = v1.MergeLinterSettings(base.settings, loaded.policy.LinterSettings, loaded.presence, true) - result.chain = append(base.chain, loaded.path) + result.chain = append(base.chain, loaded.canonical) } else { result.linters = v1.BaseLinterSection(loaded.policy.Linters, loaded.presence) result.settings = cloneSettings(loaded.policy.LinterSettings) - result.chain = []string{loaded.path} + result.chain = []string{loaded.canonical} } r.resolved[key] = result return result, nil } func (r *Resolver) resolveBreakingFile(ctx context.Context, path, boundary string, graphKey string, expandEnvironment bool, stack []string) (resolvedSection, error) { - loaded, err := r.load(path, expandEnvironment) + loaded, err := r.load(path, boundary, expandEnvironment) if err != nil { return resolvedSection{}, err } - key := resolveKey{path: loaded.canonical, section: v1.PolicySectionBreaking, graph: graphKey, boundary: boundary, expandEnvironment: loaded.expandEnvironment} + key := resolveKey{path: loaded.path, section: v1.PolicySectionBreaking, graph: graphKey, boundary: boundary, expandEnvironment: loaded.expandEnvironment} if cached, ok := r.resolved[key]; ok { if len(stack)+len(cached.chain) > maxExtendsDepth { return resolvedSection{}, fmt.Errorf("policy extends exceeds maximum depth %d at %s", maxExtendsDepth, path) @@ -246,16 +244,16 @@ func (r *Resolver) resolveBreakingFile(ctx context.Context, path, boundary strin return resolvedSection{}, fmt.Errorf("%s: breaking.extends %q resolved to %s: %w", loaded.path, loaded.policy.Breaking.Extends, basePath, err) } result.breaking = v1.MergeBreakingSection(base.breaking, loaded.policy.Breaking, loaded.presence, true) - result.chain = append(base.chain, loaded.path) + result.chain = append(base.chain, loaded.canonical) } else { result.breaking = v1.BaseBreakingSection(loaded.policy.Breaking, loaded.presence) - result.chain = []string{loaded.path} + result.chain = []string{loaded.canonical} } r.resolved[key] = result return result, nil } -func (r *Resolver) load(path string, expandEnvironment bool) (loadedPolicy, error) { +func (r *Resolver) load(path, boundary string, expandEnvironment bool) (loadedPolicy, error) { absolute, err := filepath.Abs(path) if err != nil { return loadedPolicy{}, fmt.Errorf("policy source %s: Abs: %w", path, err) @@ -264,11 +262,15 @@ func (r *Resolver) load(path string, expandEnvironment bool) (loadedPolicy, erro if err != nil { return loadedPolicy{}, fmt.Errorf("policy source %s: EvalSymlinks: %w", path, err) } - key := loadKey{path: canonical, expandEnvironment: expandEnvironment} + key := loadKey{path: absolute + "\x00" + boundary, expandEnvironment: expandEnvironment} if cached, ok := r.loaded[key]; ok { return cached, nil } - raw, err := os.ReadFile(canonical) + relative, err := filepath.Rel(boundary, absolute) + if err != nil || !filepath.IsLocal(relative) { + return loadedPolicy{}, fmt.Errorf("policy source %s is outside allowed policy root %s", path, boundary) + } + raw, err := sourceview.ReadLocal(context.Background(), boundary, relative) if err != nil { return loadedPolicy{}, fmt.Errorf("read policy source %s: %w", canonical, err) } @@ -286,7 +288,7 @@ func (r *Resolver) load(path string, expandEnvironment bool) (loadedPolicy, erro return loadedPolicy{}, fmt.Errorf("policy source %s presence: %w", canonical, err) } result := loadedPolicy{ - path: canonical, canonical: canonical, policy: parsed, presence: presence, + path: absolute, canonical: canonical, policy: parsed, presence: presence, expandEnvironment: expandEnvironment, } r.loaded[key] = result @@ -355,7 +357,12 @@ func (r *Resolver) resolveReference(from loadedPolicy, raw string, graph modules if !within(directory, boundary) { return "", "", "", false, fmt.Errorf("reference %q selects a path outside allowed policy root %s", raw, boundary) } - info, err := os.Stat(directory) + relative, err := filepath.Rel(boundary, directory) + if err != nil { + return "", "", "", false, fmt.Errorf("Rel: %w", err) + } + resolved, err := sourceview.ResolveLocal(context.Background(), boundary, relative) + info := resolved.Info if err != nil { return "", "", "", false, fmt.Errorf("reference %q selects %s: %w", raw, directory, err) } @@ -380,7 +387,7 @@ func (r *Resolver) resolveReference(from loadedPolicy, raw string, graph modules if !within(canonical, canonicalBoundary) { return "", "", "", false, fmt.Errorf("reference %q resolves outside allowed policy root %s", raw, canonicalBoundary) } - return canonical, canonicalBoundary, raw, expandEnvironment, nil + return directory, boundary, raw, expandEnvironment, nil } func matchModuleReference(reference v1.PolicyReference, graph modules.PolicyGraph) (string, string, error) { diff --git a/internal/protosource/package.go b/internal/protosource/package.go new file mode 100644 index 00000000..14e75671 --- /dev/null +++ b/internal/protosource/package.go @@ -0,0 +1,57 @@ +package protosource + +import ( + "bytes" + "strings" + "text/scanner" +) + +// Package lexes only top-level package declarations. Scanner tokens keep +// strings and comments opaque, so an option containing "package" cannot select +// a file. It intentionally does not validate unrelated message definitions. +func Package(raw []byte) string { + var lex scanner.Scanner + lex.Init(bytes.NewReader(raw)) + lex.Mode = scanner.ScanIdents | scanner.ScanStrings | scanner.ScanChars | scanner.ScanComments | scanner.SkipComments + lex.Error = func(*scanner.Scanner, string) {} + depth := 0 + start := true + for token := lex.Scan(); token != scanner.EOF; token = lex.Scan() { + if depth == 0 && start && token == scanner.Ident && lex.TokenText() == "package" { + var parts []string + for { + if lex.Scan() != scanner.Ident { + return "" + } + parts = append(parts, lex.TokenText()) + switch lex.Scan() { + case ';': + return strings.Join(parts, ".") + case '.': + continue + default: + return "" + } + } + } + switch token { + case '{', '(', '[': + depth++ + start = false + case '}', ')', ']': + if depth > 0 { + depth-- + } + start = depth == 0 + case ';': + if depth == 0 { + start = true + } + default: + if depth == 0 { + start = false + } + } + } + return "" +} diff --git a/internal/protosource/package_test.go b/internal/protosource/package_test.go new file mode 100644 index 00000000..2cd8845d --- /dev/null +++ b/internal/protosource/package_test.go @@ -0,0 +1,24 @@ +package protosource + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestSourcePackageReadsOnlyDeclaration(t *testing.T) { + t.Parallel() + for _, tt := range []struct{ name, source, want string }{ + {name: "simple", source: "syntax = \"proto3\"; package demo.v1; message M {}", want: "demo.v1"}, + {name: "comments", source: "/*package fake;*/ syntax='proto3'; //package wrong;\npackage demo /*comment*/ . v1 ;", want: "demo.v1"}, + {name: "string option", source: "syntax=\"proto3\"; option java_package=\"package wrong;\"; package demo.v1;", want: "demo.v1"}, + {name: "field name", source: "message M { string package = 1; }"}, + {name: "late declaration", source: "message M {} package demo.v1;", want: "demo.v1"}, + {name: "broken unselected body", source: "package other.v1; message {", want: "other.v1"}, + {name: "malformed package", source: "package demo..v1;"}, + {name: "empty", source: ""}, + {name: "byte order mark", source: "\ufeffpackage demo.v1;", want: "demo.v1"}, + } { + t.Run(tt.name, func(t *testing.T) { t.Parallel(); assert.Equal(t, tt.want, Package([]byte(tt.source))) }) + } +} diff --git a/internal/sourceview/local.go b/internal/sourceview/local.go new file mode 100644 index 00000000..bc18e523 --- /dev/null +++ b/internal/sourceview/local.go @@ -0,0 +1,195 @@ +package sourceview + +import ( + "context" + "errors" + "fmt" + "io" + "io/fs" + "os" + "path/filepath" +) + +// ErrNestedRepository marks an alias crossing into a nested local repository. +var ErrNestedRepository = errors.New("source alias crosses nested repository") + +// OpenLocal opens a logical source through a bounded local root. Closing the +// returned file also closes the root handle. +func OpenLocal(ctx context.Context, rootPath, logical string) (fs.File, error) { + return OpenLocalSelected(ctx, rootPath, logical, nil) +} + +// OpenLocalSelected validates a resolved target before opening its physical +// regular path and checks that the opened inode is the selected target. +// An optional nestedAllowed predicate must prove explicit source ownership for +// a target crossing a nested repository boundary. +func OpenLocalSelected(ctx context.Context, rootPath, logical string, selected func(Resolution) bool, nestedAllowed ...func(Resolution) bool) (fs.File, error) { + root, view, err := localView(rootPath) + if err != nil { + return nil, fmt.Errorf("localView: %w", err) + } + resolved, err := view.Resolve(ctx, filepath.ToSlash(logical)) + if err != nil { + return nil, errors.Join(fmt.Errorf("Resolve: %w", localResolutionError(logical, resolved, err)), root.Close()) + } + if err := CheckLocalResolution(root, resolved); err != nil { + permitted := errors.Is(err, ErrNestedRepository) && len(nestedAllowed) > 0 && nestedAllowed[0] != nil && nestedAllowed[0](resolved) + if !permitted { + return nil, errors.Join(err, root.Close()) + } + } + if !resolved.Info.Mode().IsRegular() { + return nil, errors.Join(ErrUnsupported, root.Close()) + } + if selected != nil && !selected(resolved) { + return nil, errors.Join(&fs.PathError{Op: "open", Path: logical, Err: fs.ErrNotExist}, root.Close()) + } + file, err := root.Open(filepath.FromSlash(resolved.Path)) + if err != nil { + return nil, errors.Join(fmt.Errorf("Open: %w", err), root.Close()) + } + opened, err := file.Stat() + if err != nil { + return nil, errors.Join(fmt.Errorf("Stat: %w", err), file.Close(), root.Close()) + } + if !os.SameFile(resolved.Info, opened) || resolved.Info.Mode() != opened.Mode() || resolved.Info.Size() != opened.Size() || !resolved.Info.ModTime().Equal(opened.ModTime()) { + return nil, errors.Join(ErrChanged, file.Close(), root.Close()) + } + if err := ctx.Err(); err != nil { + return nil, errors.Join(err, file.Close(), root.Close()) + } + return &localFile{File: file, root: root}, nil +} + +type localFile struct { + fs.File + root *os.Root +} + +func (file *localFile) Close() error { return errors.Join(file.File.Close(), file.root.Close()) } + +// ReadLocal reads a logical regular source without leaving its local root. +func ReadLocal(ctx context.Context, rootPath, logical string) (_ []byte, resultErr error) { + file, err := OpenLocal(ctx, rootPath, logical) + if err != nil { + return nil, fmt.Errorf("OpenLocal: %w", err) + } + defer func() { resultErr = errors.Join(resultErr, file.Close()) }() + info, err := file.Stat() + if err != nil { + return nil, fmt.Errorf("Stat: %w", err) + } + if !info.Mode().IsRegular() { + return nil, fmt.Errorf("source %q is not a regular file", logical) + } + data, err := io.ReadAll(file) + if err != nil { + return nil, fmt.Errorf("ReadAll: %w", err) + } + return data, nil +} + +// WalkLocal walks a bounded local tree while preserving logical source names. +func WalkLocal(ctx context.Context, rootPath, logical string, visit WalkFunc) (resultErr error) { + root, view, err := localView(rootPath) + if err != nil { + return fmt.Errorf("localView: %w", err) + } + defer func() { resultErr = errors.Join(resultErr, root.Close()) }() + return view.Walk(ctx, filepath.ToSlash(logical), func(name string, resolved Resolution, err error) error { + if err == nil { + err = CheckLocalResolution(root, resolved) + } + return visit(name, resolved, err) + }) +} + +// ResolveLocal resolves a logical source without leaving its local root. +func ResolveLocal(ctx context.Context, rootPath, logical string) (_ Resolution, resultErr error) { + root, view, err := localView(rootPath) + if err != nil { + return Resolution{}, fmt.Errorf("localView: %w", err) + } + defer func() { resultErr = errors.Join(resultErr, root.Close()) }() + resolved, err := view.Resolve(ctx, filepath.ToSlash(logical)) + if err == nil { + err = CheckLocalResolution(root, resolved) + } + return resolved, localResolutionError(logical, resolved, err) +} + +func localView(rootPath string) (*os.Root, *View, error) { + canonical, err := filepath.EvalSymlinks(rootPath) + if err != nil { + return nil, nil, fmt.Errorf("EvalSymlinks: %w", err) + } + canonical, err = filepath.Abs(canonical) + if err != nil { + return nil, nil, fmt.Errorf("Abs: %w", err) + } + root, err := os.OpenRoot(canonical) + if err != nil { + return nil, nil, fmt.Errorf("OpenRoot: %w", err) + } + requested, err := filepath.Abs(rootPath) + if err != nil { + return nil, nil, errors.Join(fmt.Errorf("Abs: %w", err), root.Close()) + } + original, err := os.Stat(requested) + if err != nil { + return nil, nil, errors.Join(fmt.Errorf("Stat: %w", err), root.Close()) + } + opened, err := root.Stat(".") + if err != nil { + return nil, nil, errors.Join(fmt.Errorf("Stat: %w", err), root.Close()) + } + if !os.SameFile(original, opened) { + return nil, nil, errors.Join(ErrChanged, root.Close()) + } + view, err := NewLocal(root.FS(), canonical, requested) + if err != nil { + return nil, nil, errors.Join(fmt.Errorf("NewLocal: %w", err), root.Close()) + } + return root, view, nil +} + +func localResolutionError(logical string, resolved Resolution, err error) error { + if errors.Is(err, fs.ErrNotExist) && len(resolved.Links) > 0 { + return fmt.Errorf("source alias %q is dangling: %v: %w", logical, err, ErrUnsupported) + } + return err +} + +// CheckLocalResolution rejects alias hops and targets inside nested repositories. +// The boundary root's own Git marker is allowed. Graph readers may authorize a +// target through a separately declared source boundary using OpenLocalSelected. +func CheckLocalResolution(root *os.Root, resolved Resolution) error { + if len(resolved.Links) == 0 { + return nil + } + directories := []string{filepath.Dir(filepath.FromSlash(resolved.Path))} + if resolved.Info != nil && resolved.Info.IsDir() { + directories[0] = filepath.FromSlash(resolved.Path) + } + for _, link := range resolved.Links { + directories = append(directories, filepath.Dir(filepath.FromSlash(link.Path))) + } + seen := make(map[string]bool) + for _, directory := range directories { + for current := directory; current != "."; current = filepath.Dir(current) { + if seen[current] { + break + } + seen[current] = true + _, err := root.Lstat(filepath.Join(current, ".git")) + if errors.Is(err, os.ErrNotExist) { + continue + } + if err != nil { + return fmt.Errorf("Lstat: %w", err) + } + return fmt.Errorf("source target %q crosses nested repository %q: %w", resolved.Path, current, errors.Join(ErrUnsupported, ErrNestedRepository)) + } + } + return nil +} diff --git a/internal/sourceview/local_test.go b/internal/sourceview/local_test.go new file mode 100644 index 00000000..b233b734 --- /dev/null +++ b/internal/sourceview/local_test.go @@ -0,0 +1,35 @@ +package sourceview + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestOpenLocalSelectedRejectsChangedPhysicalTarget(t *testing.T) { + t.Parallel() + root := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(root, "file.proto"), []byte("original"), 0o600)) + require.NoError(t, os.WriteFile(filepath.Join(root, "replacement.proto"), []byte("modified"), 0o600)) + file, err := OpenLocalSelected(t.Context(), root, "file.proto", func(Resolution) bool { + require.NoError(t, os.Rename(filepath.Join(root, "replacement.proto"), filepath.Join(root, "file.proto"))) + return true + }) + require.ErrorIs(t, err, ErrChanged) + require.Nil(t, file) +} + +func TestOpenLocalRejectsNestedRepositoryAlias(t *testing.T) { + t.Parallel() + root := t.TempDir() + require.NoError(t, os.MkdirAll(filepath.Join(root, "nested/.git"), 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(root, "nested/file.proto"), []byte("nested source"), 0o600)) + require.NoError(t, os.Symlink("nested/file.proto", filepath.Join(root, "alias.proto"))) + file, err := OpenLocal(t.Context(), root, "alias.proto") + if file != nil { + require.NoError(t, file.Close()) + } + require.ErrorIs(t, err, ErrUnsupported) +} diff --git a/internal/sourceview/view.go b/internal/sourceview/view.go new file mode 100644 index 00000000..432c2d96 --- /dev/null +++ b/internal/sourceview/view.go @@ -0,0 +1,341 @@ +// Package sourceview resolves logical source names within a bounded filesystem. +package sourceview + +import ( + "context" + "errors" + "fmt" + "io/fs" + "os" + "path" + "path/filepath" + "slices" + "strings" +) + +var ( + // ErrOutsideRoot marks a pointer that leaves the filesystem boundary. + ErrOutsideRoot = errors.New("source target is outside root") + // ErrCycle marks an active link or directory expansion cycle. + ErrCycle = errors.New("source link cycle") + // ErrUnsupported marks an unsupported pointer spelling or file type. + ErrUnsupported = errors.New("unsupported source target") + // ErrChanged marks a target replaced between resolution and opening. + ErrChanged = errors.New("source target changed during open") +) + +// View resolves relative logical names against a borrowed, bounded filesystem. +// Its filesystem should implement fs.ReadLinkFS when it contains symbolic links. +// A local filesystem must enforce its own boundary, such as os.Root.FS. +// View does not close the filesystem and is safe for concurrent use when it is. +type View struct { + fsys fs.FS + absoluteRoots []string +} + +// Resolution describes a target and the physical links followed to reach it. +// Path is a relative physical path in the supplied filesystem. On failure it +// describes the last inspected target, and Info may be nil. +type Resolution struct { + Path string + Info fs.FileInfo + Links []Link +} + +// Link preserves a physical link's original target and its unfollowed metadata. +type Link struct { + Path string + Target string + Info fs.FileInfo +} + +// WalkFunc receives logical names, their resolution, and any resolution or +// directory-reading error. Returning nil ignores the error and skips that entry. +// fs.SkipDir and fs.SkipAll have the same meanings as in fs.WalkDir. +type WalkFunc func(logical string, resolved Resolution, err error) error + +// New borrows fsys and rejects all absolute symbolic link targets. +func New(fsys fs.FS) *View { + return &View{fsys: fsys} +} + +// NewLocal borrows fsys and accepts absolute links beneath canonicalRoot. +// canonicalRoot must be the absolute, canonical OS path of fsys's boundary. +// The caller must canonicalize it before opening the bounded filesystem. +// rootAliases are alternative absolute spellings that the caller has verified +// identify the same physical root. Targets are never canonicalized on the host. +func NewLocal(fsys fs.FS, canonicalRoot string, rootAliases ...string) (*View, error) { + roots := append([]string{canonicalRoot}, rootAliases...) + for index, root := range roots { + if !filepath.IsAbs(root) || strings.ContainsRune(root, '\x00') { + return nil, &fs.PathError{Op: "NewLocal", Path: root, Err: fs.ErrInvalid} + } + roots[index] = filepath.ToSlash(filepath.Clean(root)) + } + return &View{fsys: fsys, absoluteRoots: roots}, nil +} + +// Resolve follows links component by component without cleaning away dots +// before preceding links have resolved. Relative parents cannot leave the root. +// Link traces are also returned for dangling or rejected targets. +func (v *View) Resolve(ctx context.Context, logical string) (Resolution, error) { + resolved := Resolution{Path: "."} + err := ctx.Err() + if err != nil { + return resolved, &fs.PathError{Op: "Resolve", Path: logical, Err: err} + } + if logical == "" || v.fsys == nil { + return resolved, &fs.PathError{Op: "Resolve", Path: logical, Err: fs.ErrInvalid} + } + err = validatePointer(logical) + if err != nil { + return resolved, &fs.PathError{Op: "Resolve", Path: logical, Err: err} + } + if strings.HasPrefix(logical, "/") { + return resolved, &fs.PathError{Op: "Resolve", Path: logical, Err: ErrOutsideRoot} + } + err = v.resolveComponents(ctx, ".", strings.Split(logical, "/"), make(map[string]bool), &resolved) + if err == nil { + err = ctx.Err() + } + if err != nil { + return resolved, &fs.PathError{Op: "Resolve", Path: logical, Err: err} + } + return resolved, nil +} + +func (v *View) resolveComponents(ctx context.Context, base string, components []string, active map[string]bool, resolved *Resolution) error { + err := ctx.Err() + if err != nil { + return err + } + info, err := fs.Lstat(v.fsys, base) + resolved.Path, resolved.Info = base, info + if err != nil { + return fmt.Errorf("Lstat: %w", err) + } + if !info.IsDir() { + return ErrChanged + } + current := base + for _, component := range components { + err = ctx.Err() + if err != nil { + return err + } + if !info.IsDir() { + return fs.ErrInvalid + } + switch component { + case "", ".": + continue + case "..": + if current == "." { + return ErrOutsideRoot + } + current = path.Dir(current) + default: + current = joinName(current, component) + } + info, err = fs.Lstat(v.fsys, current) + resolved.Path, resolved.Info = current, info + if err != nil { + return fmt.Errorf("Lstat: %w", err) + } + if info.Mode()&fs.ModeSymlink != 0 { + linkPath := current + target, readErr := fs.ReadLink(v.fsys, linkPath) + resolved.Links = append(resolved.Links, Link{Path: linkPath, Target: target, Info: info}) + if readErr != nil { + return fmt.Errorf("ReadLink: %w", readErr) + } + if active[linkPath] { + return ErrCycle + } + err = validatePointer(target) + if err != nil { + return err + } + if target == "" { + return ErrUnsupported + } + base = path.Dir(linkPath) + if strings.HasPrefix(target, "/") { + target, err = v.relativeAbsoluteTarget(target) + if err != nil { + return err + } + base = "." + } + active[linkPath] = true + err = v.resolveComponents(ctx, base, strings.Split(target, "/"), active, resolved) + delete(active, linkPath) + if err != nil { + return err + } + current, info = resolved.Path, resolved.Info + } + if !info.IsDir() && !info.Mode().IsRegular() { + return ErrUnsupported + } + } + return nil +} + +// relativeAbsoluteTarget deliberately retains dot components. Cleaning first +// would change the meaning of a parent component following a directory alias. +func (v *View) relativeAbsoluteTarget(target string) (string, error) { + for _, root := range v.absoluteRoots { + if target == root { + return ".", nil + } + prefix := strings.TrimRight(root, "/") + "/" + if strings.HasPrefix(target, prefix) { + return strings.TrimPrefix(target, prefix), nil + } + } + return "", ErrOutsideRoot +} + +func validatePointer(target string) error { + if strings.ContainsAny(target, "\\\x00") || strings.HasPrefix(target, "//") { + return ErrUnsupported + } + if len(target) >= 2 && target[1] == ':' { + return ErrUnsupported + } + return nil +} + +// Open returns a resolved regular file. It checks the opened metadata against +// the resolution and closes the opened file if validation fails. Native OS +// metadata permits an identity check; synthetic metadata is checked by type, +// size, and modification time. The caller owns the returned file. +func (v *View) Open(ctx context.Context, logical string) (fs.File, error) { + resolved, err := v.Resolve(ctx, logical) + if err != nil { + return nil, fmt.Errorf("Resolve: %w", err) + } + if !resolved.Info.Mode().IsRegular() { + return nil, &fs.PathError{Op: "Open", Path: logical, Err: ErrUnsupported} + } + err = ctx.Err() + if err != nil { + return nil, &fs.PathError{Op: "Open", Path: logical, Err: err} + } + file, err := v.fsys.Open(resolved.Path) + if err != nil { + return nil, fmt.Errorf("Open: %w", err) + } + info, err := file.Stat() + if err != nil { + return nil, closeWithError(file, fmt.Errorf("Stat: %w", err)) + } + err = ctx.Err() + if err != nil { + return nil, closeWithError(file, &fs.PathError{Op: "Open", Path: logical, Err: err}) + } + if !sameTarget(resolved.Info, info) { + return nil, closeWithError(file, &fs.PathError{Op: "Open", Path: logical, Err: ErrChanged}) + } + return file, nil +} + +func sameTarget(expected, actual fs.FileInfo) bool { + if actual == nil || !actual.Mode().IsRegular() || expected.Mode() != actual.Mode() { + return false + } + if os.SameFile(expected, expected) && !os.SameFile(expected, actual) { + return false + } + return expected.Size() == actual.Size() && expected.ModTime().Equal(actual.ModTime()) +} + +func closeWithError(file fs.File, err error) error { + closeErr := file.Close() + if closeErr != nil { + return errors.Join(err, fmt.Errorf("Close: %w", closeErr)) + } + return err +} + +// Walk visits logicalRoot and sorted descendants, following directory aliases +// while preserving logical prefixes. An active physical directory is a cycle, +// but distinct aliases visited in separate branches are expanded independently. +// Resolution and directory-reading failures are handed to fn for classification. +func (v *View) Walk(ctx context.Context, logicalRoot string, fn WalkFunc) error { + if fn == nil { + return &fs.PathError{Op: "Walk", Path: logicalRoot, Err: fs.ErrInvalid} + } + err := v.walk(ctx, logicalRoot, fn, make(map[string]bool)) + if errors.Is(err, fs.SkipDir) || errors.Is(err, fs.SkipAll) { + return nil + } + return err +} + +func (v *View) walk(ctx context.Context, logical string, fn WalkFunc, active map[string]bool) error { + err := ctx.Err() + if err != nil { + return &fs.PathError{Op: "Walk", Path: logical, Err: err} + } + resolved, resolveErr := v.Resolve(ctx, logical) + err = ctx.Err() + if err != nil { + return &fs.PathError{Op: "Walk", Path: logical, Err: err} + } + if resolveErr == nil && resolved.Info.IsDir() && active[resolved.Path] { + resolveErr = &fs.PathError{Op: "Walk", Path: logical, Err: ErrCycle} + } + err = fn(logical, resolved, resolveErr) + if errors.Is(err, fs.SkipDir) && resolved.Info != nil && resolved.Info.IsDir() { + return nil + } + if err != nil { + return err + } + if resolveErr != nil || !resolved.Info.IsDir() { + return nil + } + err = ctx.Err() + if err != nil { + return &fs.PathError{Op: "Walk", Path: logical, Err: err} + } + entries, err := fs.ReadDir(v.fsys, resolved.Path) + contextErr := ctx.Err() + if contextErr != nil { + return &fs.PathError{Op: "Walk", Path: logical, Err: contextErr} + } + if err != nil { + err = fn(logical, resolved, fmt.Errorf("ReadDir: %w", err)) + if errors.Is(err, fs.SkipDir) { + return nil + } + return err + } + slices.SortFunc(entries, func(a, b fs.DirEntry) int { return strings.Compare(a.Name(), b.Name()) }) + active[resolved.Path] = true + defer delete(active, resolved.Path) + for _, entry := range entries { + name := entry.Name() + if !fs.ValidPath(name) || strings.Contains(name, "/") || name == "." { + err = fn(joinName(logical, name), Resolution{}, &fs.PathError{Op: "Walk", Path: name, Err: fs.ErrInvalid}) + } else { + err = v.walk(ctx, joinName(logical, name), fn, active) + } + if errors.Is(err, fs.SkipDir) { + return nil + } + if err != nil { + return err + } + } + return nil +} + +func joinName(parent, name string) string { + if parent == "." { + return name + } + return strings.TrimRight(parent, "/") + "/" + name +} diff --git a/internal/sourceview/view_test.go b/internal/sourceview/view_test.go new file mode 100644 index 00000000..ffbbae79 --- /dev/null +++ b/internal/sourceview/view_test.go @@ -0,0 +1,494 @@ +package sourceview + +import ( + "context" + "errors" + "io" + "io/fs" + "os" + "path/filepath" + "strings" + "testing" + "testing/fstest" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestResolvePreservesComponentOrderAndLinkTrace(t *testing.T) { + t.Parallel() + view := New(fstest.MapFS{ + "real/nested/file.proto": {Data: []byte("message File {}")}, + "real/sibling.proto": {Data: []byte("message Sibling {}")}, + "alias": {Mode: fs.ModeSymlink, Data: []byte("real/nested")}, + "chain": {Mode: fs.ModeSymlink, Data: []byte("alias/file.proto")}, + "chain2": {Mode: fs.ModeSymlink, Data: []byte("chain")}, + "relative": {Mode: fs.ModeSymlink, Data: []byte("alias/../sibling.proto")}, + }) + tests := []struct { + name string + input string + path string + links []string + }{ + {name: "file chain", input: "chain2", path: "real/nested/file.proto", links: []string{"chain2", "chain", "alias"}}, + {name: "intermediate directory", input: "alias/./file.proto", path: "real/nested/file.proto", links: []string{"alias"}}, + {name: "parent after alias", input: "alias/../sibling.proto", path: "real/sibling.proto", links: []string{"alias"}}, + {name: "parent inside target", input: "relative", path: "real/sibling.proto", links: []string{"relative", "alias"}}, + {name: "root", input: ".", path: "."}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + resolved, err := view.Resolve(t.Context(), tt.input) + require.NoError(t, err) + assert.Equal(t, tt.path, resolved.Path) + require.NotNil(t, resolved.Info) + var paths []string + for _, link := range resolved.Links { + paths = append(paths, link.Path) + assert.NotEmpty(t, link.Target) + assert.Equal(t, fs.ModeSymlink, link.Info.Mode().Type()) + } + assert.Equal(t, tt.links, paths) + }) + } + file, err := view.Open(t.Context(), "chain2") + require.NoError(t, err) + data, err := io.ReadAll(file) + require.NoError(t, err) + require.NoError(t, file.Close()) + assert.Equal(t, "message File {}", string(data)) +} + +func TestResolveLinkCyclesUsePhysicalPaths(t *testing.T) { + t.Parallel() + view := New(fstest.MapFS{ + "one": {Mode: fs.ModeSymlink, Data: []byte("two")}, + "two": {Mode: fs.ModeSymlink, Data: []byte("one")}, + "root": {Mode: fs.ModeSymlink, Data: []byte("inner/leaf")}, + "inner/leaf": {Mode: fs.ModeSymlink, Data: []byte("inner/leaf")}, + "inner/inner/leaf": {Data: []byte("same link contents are not identities")}, + "again": {Mode: fs.ModeSymlink, Data: []byte(".")}, + "regular/file.proto": {Data: []byte("file")}, + "expanding": {Mode: fs.ModeSymlink, Data: []byte("expanding/child")}, + "target-parent": {Mode: fs.ModeSymlink, Data: []byte("regular/../target-parent")}, + }) + for _, input := range []string{"one", "expanding", "target-parent"} { + _, err := view.Resolve(t.Context(), input) + require.ErrorIs(t, err, ErrCycle) + } + resolved, err := view.Resolve(t.Context(), "root") + require.NoError(t, err) + assert.Equal(t, "inner/inner/leaf", resolved.Path) + require.Len(t, resolved.Links, 2) + assert.Equal(t, resolved.Links[0].Target, resolved.Links[1].Target) + resolved, err = view.Resolve(t.Context(), "again/again/regular/file.proto") + require.NoError(t, err) + assert.Equal(t, "regular/file.proto", resolved.Path) +} + +func TestResolveRejectsUnsafeAndIrregularTargets(t *testing.T) { + t.Parallel() + tests := []struct { + name string + target string + want error + }{ + {name: "parent escape", target: "../outside", want: ErrOutsideRoot}, + {name: "parent after root alias", target: "root/../outside", want: ErrOutsideRoot}, + {name: "absolute", target: "/outside/secret", want: ErrOutsideRoot}, + {name: "drive absolute", target: "C:/outside", want: ErrUnsupported}, + {name: "drive relative", target: "C:outside", want: ErrUnsupported}, + {name: "UNC", target: "//server/share", want: ErrUnsupported}, + {name: "backslash", target: "dir\\file", want: ErrUnsupported}, + {name: "NUL", target: "dir\x00file", want: ErrUnsupported}, + {name: "dangling", target: "missing", want: fs.ErrNotExist}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + backend := &openedFS{MapFS: fstest.MapFS{ + "pointer": {Mode: fs.ModeSymlink, Data: []byte(tt.target)}, + "root": {Mode: fs.ModeSymlink, Data: []byte(".")}, + }} + view := New(backend) + resolved, err := view.Resolve(t.Context(), "pointer") + require.ErrorIs(t, err, tt.want) + require.NotEmpty(t, resolved.Links) + assert.Equal(t, tt.target, resolved.Links[0].Target) + _, err = view.Open(t.Context(), "pointer") + require.ErrorIs(t, err, tt.want) + assert.Empty(t, backend.opened, "rejected targets must not be opened") + }) + } + view := New(fstest.MapFS{ + "gitlink": {Mode: fs.ModeIrregular}, + "fifo": {Mode: fs.ModeNamedPipe}, + "file": {Data: []byte("file")}, + }) + for _, input := range []string{"gitlink", "fifo"} { + resolved, err := view.Resolve(t.Context(), input) + require.ErrorIs(t, err, ErrUnsupported) + require.NotNil(t, resolved.Info) + assert.False(t, resolved.Info.Mode().IsRegular()) + } + for _, input := range []string{"file/..", "file/.", "file/child"} { + _, err := view.Resolve(t.Context(), input) + require.Error(t, err, input) + } +} + +func TestLocalAbsolutePointersStayInsideRoot(t *testing.T) { + t.Parallel() + rootPath := t.TempDir() + canonicalRoot, err := filepath.EvalSymlinks(rootPath) + require.NoError(t, err) + require.NoError(t, os.MkdirAll(filepath.Join(rootPath, "real", "nested"), 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(rootPath, "real", "file.proto"), []byte("bounded"), 0o600)) + require.NoError(t, os.Symlink("real/nested", filepath.Join(rootPath, "dir"))) + absTarget := canonicalRoot + "/dir/../file.proto" + require.NoError(t, os.Symlink(absTarget, filepath.Join(rootPath, "absolute"))) + require.NoError(t, os.Symlink(canonicalRoot, filepath.Join(rootPath, "root"))) + outside := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(outside, "secret"), []byte("external"), 0o600)) + require.NoError(t, os.Symlink(filepath.Join(outside, "secret"), filepath.Join(rootPath, "external"))) + require.NoError(t, os.Symlink(canonicalRoot+"/../external", filepath.Join(rootPath, "parent-escape"))) + root, err := os.OpenRoot(rootPath) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, root.Close()) }) + view, err := NewLocal(root.FS(), canonicalRoot) + require.NoError(t, err) + resolved, err := view.Resolve(t.Context(), "absolute") + require.NoError(t, err) + assert.Equal(t, "real/file.proto", resolved.Path) + assert.Equal(t, absTarget, resolved.Links[0].Target) + file, err := view.Open(t.Context(), "root/absolute") + require.NoError(t, err) + data, err := io.ReadAll(file) + require.NoError(t, err) + require.NoError(t, file.Close()) + assert.Equal(t, "bounded", string(data)) + for _, input := range []string{"external", "parent-escape"} { + _, err = view.Open(t.Context(), input) + require.ErrorIs(t, err, ErrOutsideRoot) + } + _, err = New(root.FS()).Resolve(t.Context(), "absolute") + require.ErrorIs(t, err, ErrOutsideRoot) + _, err = NewLocal(root.FS(), "relative") + require.ErrorIs(t, err, fs.ErrInvalid) + _, err = root.Stat("real/file.proto") + require.NoError(t, err, "view must not close its borrowed root") +} + +func TestLocalVerifiedRootSpellingsPreserveUncleanedPointer(t *testing.T) { + t.Parallel() + backend := &openedFS{MapFS: fstest.MapFS{ + "real/nested": {Mode: fs.ModeDir}, + "real/file": {Data: []byte("bounded")}, + "dir": {Mode: fs.ModeSymlink, Data: []byte("real/nested")}, + "pointer": {Mode: fs.ModeSymlink, Data: []byte("/var/source/dir/../file")}, + }} + view, err := NewLocal(backend, "/private/var/source", "/var/source") + require.NoError(t, err) + resolved, err := view.Resolve(t.Context(), "pointer") + require.NoError(t, err) + assert.Equal(t, "real/file", resolved.Path) + assert.Equal(t, "/var/source/dir/../file", resolved.Links[0].Target) + assert.Empty(t, backend.opened) + _, err = NewLocal(backend, "/private/var/source", "relative") + require.ErrorIs(t, err, fs.ErrInvalid) +} + +func TestWalkKeepsDistinctAliasesAndSortedLogicalNames(t *testing.T) { + t.Parallel() + view := New(unsortedFS{MapFS: fstest.MapFS{ + "real/z.proto": {Data: []byte("z")}, + "real/a.proto": {Data: []byte("a")}, + "alias-one": {Mode: fs.ModeSymlink, Data: []byte("real")}, + "alias-two": {Mode: fs.ModeSymlink, Data: []byte("real")}, + }}) + var visited []string + err := view.Walk(t.Context(), ".", func(logical string, resolved Resolution, err error) error { + require.NoError(t, err) + visited = append(visited, logical+"="+resolved.Path) + return nil + }) + require.NoError(t, err) + assert.Equal(t, []string{ + ".=.", + "alias-one=real", "alias-one/a.proto=real/a.proto", "alias-one/z.proto=real/z.proto", + "alias-two=real", "alias-two/a.proto=real/a.proto", "alias-two/z.proto=real/z.proto", + "real=real", "real/a.proto=real/a.proto", "real/z.proto=real/z.proto", + }, visited) + visited = nil + err = view.Walk(t.Context(), "alias-one", func(logical string, resolved Resolution, err error) error { + require.NoError(t, err) + visited = append(visited, logical) + return nil + }) + require.NoError(t, err) + assert.Equal(t, []string{"alias-one", "alias-one/a.proto", "alias-one/z.proto"}, visited) +} + +func TestWalkReportsAuxiliaryErrorsAndAncestorDirectoryCycles(t *testing.T) { + t.Parallel() + view := New(fstest.MapFS{ + "dir/file.proto": {Data: []byte("file")}, + "dir/up": {Mode: fs.ModeSymlink, Data: []byte("..")}, + "unused": {Mode: fs.ModeSymlink, Data: []byte("/external")}, + "skipped/bad": {Mode: fs.ModeSymlink, Data: []byte("/external")}, + }) + var visited []string + var failed []string + err := view.Walk(t.Context(), ".", func(logical string, resolved Resolution, err error) error { + visited = append(visited, logical) + if err != nil { + failed = append(failed, logical) + require.NotEmpty(t, resolved.Links) + if logical == "dir/up" { + require.ErrorIs(t, err, ErrCycle) + } else { + require.ErrorIs(t, err, ErrOutsideRoot) + } + return nil + } + if logical == "skipped" { + return fs.SkipDir + } + return nil + }) + require.NoError(t, err) + assert.Equal(t, []string{".", "dir", "dir/file.proto", "dir/up", "skipped", "unused"}, visited) + assert.Equal(t, []string{"dir/up", "unused"}, failed) + err = view.Walk(t.Context(), ".", func(_ string, _ Resolution, err error) error { return err }) + require.ErrorIs(t, err, ErrCycle) +} + +func TestWalkSkipControlsAndContext(t *testing.T) { + t.Parallel() + view := New(fstest.MapFS{"a": {Data: []byte("a")}, "b": {Data: []byte("b")}}) + for _, skip := range []error{fs.SkipDir, fs.SkipAll} { + var visited []string + err := view.Walk(t.Context(), ".", func(logical string, _ Resolution, _ error) error { + visited = append(visited, logical) + if logical == "a" { + return skip + } + return nil + }) + require.NoError(t, err) + assert.Equal(t, []string{".", "a"}, visited) + } + ctx, cancel := context.WithCancel(t.Context()) + cancel() + _, err := view.Resolve(ctx, "a") + require.ErrorIs(t, err, context.Canceled) + _, err = view.Open(ctx, "a") + require.ErrorIs(t, err, context.Canceled) + err = view.Walk(ctx, ".", func(_ string, _ Resolution, _ error) error { + t.Fatal("canceled walk invoked callback") + return nil + }) + require.ErrorIs(t, err, context.Canceled) + ctx, cancel = context.WithCancel(t.Context()) + err = view.Walk(ctx, ".", func(logical string, _ Resolution, _ error) error { + if logical == "." { + cancel() + } + return nil + }) + require.ErrorIs(t, err, context.Canceled) +} + +func TestWalkSkipDirOnDirectoryAliasKeepsSiblings(t *testing.T) { + t.Parallel() + view := New(fstest.MapFS{ + "a-alias": {Mode: fs.ModeSymlink, Data: []byte("directory")}, + "b.proto": {Data: []byte("b")}, + "directory/a": {Data: []byte("a")}, + }) + var visited []string + err := view.Walk(t.Context(), ".", func(logical string, _ Resolution, err error) error { + require.NoError(t, err) + visited = append(visited, logical) + if logical == "a-alias" { + return fs.SkipDir + } + return nil + }) + require.NoError(t, err) + assert.Equal(t, []string{".", "a-alias", "b.proto", "directory", "directory/a"}, visited) +} + +func TestCancellationDuringFilesystemOperationCannotBeIgnored(t *testing.T) { + t.Parallel() + ctx, cancel := context.WithCancel(t.Context()) + backend := cancelFS{MapFS: fstest.MapFS{"file": {Data: []byte("file")}}, cancel: cancel} + _, err := New(backend).Resolve(ctx, "file") + require.ErrorIs(t, err, context.Canceled) + ctx, cancel = context.WithCancel(t.Context()) + backend.cancel = cancel + err = New(backend).Walk(ctx, "file", func(_ string, _ Resolution, _ error) error { + t.Fatal("walk must propagate cancellation before callback") + return nil + }) + require.ErrorIs(t, err, context.Canceled) +} + +func TestOpenRejectsReplacementAndClosesOwnedFiles(t *testing.T) { + t.Parallel() + backend := &replacedFS{MapFS: fstest.MapFS{"file": {Data: []byte("before")}}} + _, err := New(backend).Open(t.Context(), "file") + require.ErrorIs(t, err, ErrChanged) + assert.True(t, backend.closed) + view := New(fstest.MapFS{"directory": {Mode: fs.ModeDir}}) + _, err = view.Open(t.Context(), "directory") + require.ErrorIs(t, err, ErrUnsupported) +} + +func TestOpenRejectsNativeIdentityReplacementWithMatchingMetadata(t *testing.T) { + t.Parallel() + directory := t.TempDir() + stamp := time.Unix(123456789, 0) + for name, data := range map[string]string{"file": "before", "replacement": "after!"} { + filename := filepath.Join(directory, name) + require.NoError(t, os.WriteFile(filename, []byte(data), 0o600)) + require.NoError(t, os.Chtimes(filename, stamp, stamp)) + } + root, err := os.OpenRoot(directory) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, root.Close()) }) + backend := &nativeReplacementFS{ReadLinkFS: root.FS().(fs.ReadLinkFS), directory: directory} + _, err = New(backend).Open(t.Context(), "file") + require.ErrorIs(t, err, ErrChanged) + require.NotNil(t, backend.opened) + assert.True(t, backend.opened.closed) +} + +func TestOpenStatFailureJoinsCloseError(t *testing.T) { + t.Parallel() + statErr := errors.New("stat failure") + closeErr := errors.New("close failure") + backend := &failedStatFS{MapFS: fstest.MapFS{"file": {Data: []byte("file")}}, statErr: statErr, closeErr: closeErr} + _, err := New(backend).Open(t.Context(), "file") + require.ErrorIs(t, err, statErr) + require.ErrorIs(t, err, closeErr) + assert.True(t, backend.closed) +} + +type openedFS struct { + fstest.MapFS + opened []string +} + +func (f *openedFS) Open(name string) (fs.File, error) { + f.opened = append(f.opened, name) + return f.MapFS.Open(name) +} + +type unsortedFS struct{ fstest.MapFS } + +func (f unsortedFS) ReadDir(name string) ([]fs.DirEntry, error) { + entries, err := f.MapFS.ReadDir(name) + if err != nil { + return nil, err + } + for i, j := 0, len(entries)-1; i < j; i, j = i+1, j-1 { + entries[i], entries[j] = entries[j], entries[i] + } + return entries, nil +} + +type cancelFS struct { + fstest.MapFS + cancel context.CancelFunc +} + +func (f cancelFS) Lstat(name string) (fs.FileInfo, error) { + info, err := f.MapFS.Lstat(name) + if name == "file" { + f.cancel() + } + return info, err +} + +type replacedFS struct { + fstest.MapFS + closed bool +} + +func (f *replacedFS) Open(_ string) (fs.File, error) { + return &replacedFile{Reader: strings.NewReader("after"), owner: f}, nil +} + +type replacedFile struct { + *strings.Reader + owner *replacedFS +} + +func (f *replacedFile) Close() error { + f.owner.closed = true + return nil +} + +func (f *replacedFile) Stat() (fs.FileInfo, error) { + return replacementInfo{}, nil +} + +type replacementInfo struct{} + +func (replacementInfo) Name() string { return "file" } +func (replacementInfo) Size() int64 { return 5 } +func (replacementInfo) Mode() fs.FileMode { return 0 } +func (replacementInfo) ModTime() time.Time { return time.Time{} } +func (replacementInfo) IsDir() bool { return false } +func (replacementInfo) Sys() any { return nil } + +type nativeReplacementFS struct { + fs.ReadLinkFS + directory string + opened *ownedFile +} + +func (f *nativeReplacementFS) Open(name string) (fs.File, error) { + err := os.Rename(filepath.Join(f.directory, "replacement"), filepath.Join(f.directory, name)) + if err != nil { + return nil, err + } + file, err := f.ReadLinkFS.Open(name) + if err != nil { + return nil, err + } + f.opened = &ownedFile{File: file} + return f.opened, nil +} + +type ownedFile struct { + fs.File + closed bool +} + +func (f *ownedFile) Close() error { + f.closed = true + return f.File.Close() +} + +type failedStatFS struct { + fstest.MapFS + statErr error + closeErr error + closed bool +} + +func (f *failedStatFS) Open(_ string) (fs.File, error) { return failedStatFile{owner: f}, nil } + +type failedStatFile struct{ owner *failedStatFS } + +func (f failedStatFile) Stat() (fs.FileInfo, error) { return nil, f.owner.statErr } +func (f failedStatFile) Read(_ []byte) (int, error) { return 0, io.EOF } +func (f failedStatFile) Close() error { + f.owner.closed = true + return f.owner.closeErr +} diff --git a/internal/workspace/discovery.go b/internal/workspace/discovery.go index ed1311f6..90ce6517 100644 --- a/internal/workspace/discovery.go +++ b/internal/workspace/discovery.go @@ -2,11 +2,15 @@ package workspace import ( + "context" + "errors" "fmt" "io/fs" "os" "path/filepath" "strings" + + "github.com/easyp-tech/easyp/internal/sourceview" ) // Boundary finds the nearest Git repository, or the outermost EasyP ancestor @@ -24,13 +28,13 @@ func Boundary(start string) (string, error) { } if _, err := os.Lstat(filepath.Join(dir, ".git")); err == nil { return dir, nil - } else if !os.IsNotExist(err) { + } else if !errors.Is(err, os.ErrNotExist) { return "", fmt.Errorf("Lstat: %w", err) } for _, name := range []string{"protobuf.mod", "easyp.yaml", "easyp.gen.yaml"} { if _, err := os.Stat(filepath.Join(dir, name)); err == nil { root = dir - } else if !os.IsNotExist(err) { + } else if !errors.Is(err, os.ErrNotExist) { return "", fmt.Errorf("Stat: %w", err) } } @@ -49,14 +53,16 @@ func FindUp(start, boundary, name string) (string, error) { return "", fmt.Errorf("directory %q is outside workspace %q", dir, boundary) } path := filepath.Join(dir, name) - info, err := os.Stat(path) + relative, _ := filepath.Rel(boundary, path) + resolved, err := sourceview.ResolveLocal(context.Background(), boundary, relative) + info := resolved.Info if err == nil { if !info.Mode().IsRegular() { return "", fmt.Errorf("configuration %q is not a regular file", path) } return path, nil } - if !os.IsNotExist(err) { + if !errors.Is(err, os.ErrNotExist) { return "", fmt.Errorf("Stat: %w", err) } if dir == boundary || dir == filepath.Dir(dir) { @@ -90,7 +96,7 @@ func Policy(start string) (string, error) { return path, err } var found []string - err = filepath.WalkDir(start, func(path string, entry fs.DirEntry, walkErr error) error { + err = Walk(start, func(path string, entry fs.DirEntry, walkErr error) error { if walkErr != nil { return walkErr } @@ -101,13 +107,15 @@ func Policy(start string) (string, error) { return filepath.SkipDir } candidate := filepath.Join(path, "easyp.yaml") - if info, err := os.Stat(candidate); err == nil { + relative, _ := filepath.Rel(boundary, candidate) + resolved, err := sourceview.ResolveLocal(context.Background(), boundary, relative) + if info := resolved.Info; err == nil { if !info.Mode().IsRegular() { return fmt.Errorf("configuration %q is not a regular file", candidate) } found = append(found, candidate) return filepath.SkipDir - } else if !os.IsNotExist(err) { + } else if !errors.Is(err, os.ErrNotExist) { return err } return nil diff --git a/internal/workspace/sources.go b/internal/workspace/sources.go new file mode 100644 index 00000000..693678cb --- /dev/null +++ b/internal/workspace/sources.go @@ -0,0 +1,76 @@ +package workspace + +import ( + "context" + "fmt" + "io/fs" + "path/filepath" + + "github.com/easyp-tech/easyp/internal/sourceview" +) + +// ReadFile reads workspace metadata at its logical path through a bounded root. +func ReadFile(root, path string) ([]byte, error) { + relative, err := filepath.Rel(root, path) + if err != nil { + return nil, fmt.Errorf("Rel: %w", err) + } + return sourceview.ReadLocal(context.Background(), root, relative) +} + +// ReadFileAt reads local metadata within the discovered workspace boundary. +func ReadFileAt(path string) ([]byte, error) { + boundary, err := Boundary(filepath.Dir(path)) + if err != nil { + return nil, fmt.Errorf("Boundary: %w", err) + } + return ReadFile(boundary, path) +} + +// Walk traverses logical workspace paths, including bounded directory aliases. +func Walk(root string, visit func(string, fs.DirEntry, error) error) error { + return WalkAt(root, root, visit) +} + +// WalkAt walks a logical subtree within its broader workspace boundary. +func WalkAt(boundary, root string, visit func(string, fs.DirEntry, error) error) error { + relative, err := filepath.Rel(boundary, root) + if err != nil { + return fmt.Errorf("Rel: %w", err) + } + return sourceview.WalkLocal(context.Background(), boundary, relative, func(logical string, resolved sourceview.Resolution, walkErr error) error { + path := filepath.Join(boundary, filepath.FromSlash(logical)) + if SkipDirectory(root, path) { + if resolved.Info == nil || !resolved.Info.IsDir() { + return nil + } + return fs.SkipDir + } + if walkErr != nil { + switch filepath.Base(path) { + case "easyp.yaml", "easyp.gen.yaml", "protobuf.mod", "protobuf.lock", "buf.yaml", "buf.work.yaml", "buf.lock": + default: + if filepath.Ext(path) != ".proto" && logical != filepath.ToSlash(relative) { + return nil + } + } + var entry fs.DirEntry + if resolved.Info != nil { + entry = fs.FileInfoToDirEntry(resolved.Info) + } + return visit(path, entry, walkErr) + } + if resolved.Info.IsDir() && SkipDirectory(physicalRoot(root), filepath.Join(physicalRoot(boundary), filepath.FromSlash(resolved.Path))) { + return fs.SkipDir + } + return visit(path, fs.FileInfoToDirEntry(resolved.Info), nil) + }) +} + +func physicalRoot(root string) string { + canonical, err := filepath.EvalSymlinks(root) + if err == nil { + return canonical + } + return root +} diff --git a/mcp/easypconfig/describe_test.go b/mcp/easypconfig/describe_test.go index 375ea029..d05022c2 100644 --- a/mcp/easypconfig/describe_test.go +++ b/mcp/easypconfig/describe_test.go @@ -27,6 +27,7 @@ func TestDescribeV1Files(t *testing.T) { }{ {name: "default policy", input: DescribeInput{Path: "$.linters.default"}, wantFile: v1.PolicyFile, wantPath: "linters.default", wantField: "linters.default", wantSchema: true}, {name: "generator plugin", input: DescribeInput{File: v1.GenerateFile, Path: "plugins[3].out"}, wantFile: v1.GenerateFile, wantPath: "plugins[].out", wantField: "plugins[].out", wantSchema: true}, + {name: "generation source paths", input: DescribeInput{File: v1.GenerateFile, Path: "generate.paths"}, wantFile: v1.GenerateFile, wantPath: "generate.paths", wantField: "generate.paths", wantSchema: true}, {name: "plugin binary path", input: DescribeInput{File: v1.GenerateFile, Path: "plugins[0].path"}, wantFile: v1.GenerateFile, wantPath: "plugins[].path", wantField: "plugins[].path", wantSchema: true}, {name: "custom plugin command", input: DescribeInput{File: v1.GenerateFile, Path: "plugins[0].command"}, wantFile: v1.GenerateFile, wantPath: "plugins[].command", wantField: "plugins[].command", wantSchema: true}, } @@ -89,6 +90,7 @@ func TestDescribeNotesForReservedFields(t *testing.T) { want string }{ {name: "package filter", file: v1.GenerateFile, path: "generate.packages", want: "exact protobuf packages"}, + {name: "path filter", file: v1.GenerateFile, path: "generate.paths", want: "literal module-relative"}, {name: "linter inheritance", file: v1.PolicyFile, path: "linters.extends", want: "declared-module#policy-path"}, {name: "issue path matching", file: v1.PolicyFile, path: "issues.exclude-rules[0].path", want: "relative to the easyp.yaml"}, {name: "breaking unstable", file: v1.PolicyFile, path: "breaking.ignore_unstable", want: "both comparison revisions"}, @@ -110,6 +112,34 @@ func TestDescribeNotesForReservedFields(t *testing.T) { } } +func TestDescribeGenerationPathsExplainsSelectionAndPluginOptions(t *testing.T) { + t.Parallel() + + got, err := Describe(DescribeInput{File: v1.GenerateFile, Path: "generate.paths"}) + + require.NoError(t, err) + require.Len(t, got.Fields, 1) + assert.Contains(t, got.Fields[0].Description, "intersected with generate.packages") + notes := strings.Join(got.Notes, " ") + for _, contract := range []string{"component-bounded", "even without plugins", "Module roots", "imports", "plugins.opts.paths"} { + assert.Contains(t, notes, contract) + } + require.NotEmpty(t, got.Examples) + assert.Contains(t, got.Examples[0].YAML, "paths: [mcp]") + assert.Contains(t, got.Examples[0].YAML, "paths=source_relative") +} + +func TestDescribeModuleScopedSourceSelectors(t *testing.T) { + t.Parallel() + got, err := Describe(DescribeInput{File: v1.GenerateFile, Path: "generate.modules[0].paths"}) + require.NoError(t, err) + assert.Equal(t, "generate.modules[].paths", got.SelectedPath) + require.Len(t, got.Fields, 1) + assert.Contains(t, got.Fields[0].Description, "relative to the selected module directory") + assert.Contains(t, got.Fields[0].Description, "in this module") + assert.Equal(t, "array", got.Schema["type"]) +} + func TestDescribeUsesGeneratedSchema(t *testing.T) { t.Parallel() diff --git a/mcp/easypconfig/docs.go b/mcp/easypconfig/docs.go index 1b0d6ea3..a4462740 100644 --- a/mcp/easypconfig/docs.go +++ b/mcp/easypconfig/docs.go @@ -9,7 +9,7 @@ var descriptions = map[string]map[string]string{ "modules[].source": "Exact module identity; separate v2+ lines use their matching /vN suffix.", "modules[].version": "Selected semantic version or a full commit. A commit-valued version must equal commit; major suffix and legacy +incompatible provenance are checked.", "modules[].commit": "Full 40- or 64-character hexadecimal Git commit. Tags and short SHAs are not commit pins.", - "modules[].hash": "h1: followed by standard base64 encoding of a 32-byte SHA-256 content digest. Fetch/install verify tracked contents; a sample hash is not proof of a real repository.", + "modules[].hash": "h1: followed by standard base64 encoding of a 32-byte SHA-256 content digest. Fetch/install verify the materialized logical snapshot from the pinned Git tree, including resolved internal aliases; a sample hash is not proof of a real repository.", "modules[].bsr": "BSR requests declared by this Git dependency and their recorded Git targets; frozen commands replay these bindings without calling a resolver.", "modules[].bsr[].dependency.module": "Original BSR identity, separate from the selected Git source.", "modules[].bsr[].dependency.reference": "Requested Buf label or BSR commit; preserved even when a compatibility snapshot cannot prove equivalence.", @@ -45,8 +45,12 @@ var descriptions = map[string]map[string]string{ v1.GenerateFile: { "version": "Configuration version; v1 is the only supported value.", "generate": "Select modules for this generation run.", - "generate.modules": "Module identities selected from protobuf.mod dependencies or the local workspace.", - "generate.packages": "Exact protobuf package names among selected modules. Empty means all module sources; imports remain available for compilation.", + "generate.modules": "Module identities or workspace module paths; strings select whole modules, objects add their own paths/packages intersected with global filters. Omitted selects the local module containing the generator.", + "generate.modules[].module": "Module identity or workspace-relative module directory.", + "generate.modules[].paths": "Literal module-directory-relative files or subtrees, intersected with global filters; every selector must match in this module.", + "generate.modules[].packages": "Exact protobuf packages in this module, intersected with global filters; every selector must match in this module.", + "generate.packages": "Exact protobuf package names among selected modules, intersected with generate.paths. Empty means all packages; imports remain available for compilation.", + "generate.paths": "Literal module-relative files or directory subtrees across selected modules, intersected with generate.packages and module filters. Available without generate.modules. Empty means all sources. Distinct from plugins.opts.paths, which controls plugin output layout.", "generate.managed": "Managed file and field option rules.", "plugins": "Generators executed for selected modules.", "plugins[].name": "Local or built-in plugin name.", @@ -94,6 +98,7 @@ func examplesFor(file string) []Example { case v1.GenerateFile: return []Example{ {Title: "package_selection", Description: "Requires an api.v1 package in the selected module sources.", YAML: "version: v1\ngenerate:\n packages: [api.v1]\nplugins:\n - name: go\n out: gen\n with_imports: true\n", Paths: []string{"generate.packages"}}, + {Title: "path_selection", Description: "Requires a source in the mcp import subtree of a selected module; plugin paths controls generated output layout.", YAML: "version: v1\ngenerate:\n paths: [mcp]\nplugins:\n - name: go\n out: gen\n opts: [paths=source_relative]\n", Paths: []string{"generate.paths"}}, {Title: "local_plugin", YAML: "version: v1\nplugins:\n - name: go\n out: gen/go\n opts: [paths=source_relative]\n", Paths: []string{"plugins", "plugins[]", "plugins[].name", "plugins[].out", "plugins[].opts"}}, {Title: "remote_plugin", YAML: "version: v1\nplugins:\n - remote: plugins.beta.easyp.tech/protocolbuffers/go\n version: v1.36.11\n out: gen/go\n opts: [paths=source_relative]\n", Paths: []string{"plugins", "plugins[]", "plugins[].remote", "plugins[].version"}}, {Title: "binary_path", YAML: "version: v1\nplugins:\n - path: ./tools/protoc-gen-custom\n out: gen/custom\n", Paths: []string{"plugins", "plugins[]", "plugins[].path"}}, @@ -133,7 +138,9 @@ func notesFor(file, path string) []string { case file == v1.LockFile: return []string{"Schema comes directly from v1.SchemaJSON. ParseLock adds semantic checks for duplicate sources, matching module majors and commit-valued versions. Examples contain synthetic commits/hashes, not fetched data.", "Local replace never rewrites the published lock. --frozen requires a valid complete manifest/lock graph and rejects replacements; it may download exact pinned contents but never resolves a new version. Preserve the lock when investigating a cache mismatch.", "BSR compatibility_snapshot bindings preserve the original request and Buf lock pin, but do not prove BSR revision equivalence or verify the BSR digest. Frozen commands validate metadata and reuse recorded Git targets; older locks with BSR dependencies require easyp mod tidy."} case file == v1.GenerateFile && within("generate.packages", path): - return []string{"Names match exact protobuf packages, not prefixes or file paths. Matching is across the selected modules of each project. Unknown names fail before plugins. with_imports is independent per plugin; descriptor include_imports controls exported dependencies."} + return []string{"Names match exact protobuf packages, not prefixes or file paths. Package and path filters intersect. Every selector must match an output source across the selected modules of each project, even without plugins. Unknown names fail before plugins and descriptor writes. Generation filters are not inherited. with_imports is independent per plugin; descriptor include_imports controls exported dependencies."} + case file == v1.GenerateFile && within("generate.paths", path): + return []string{"Selectors match literal module-relative file names or component-bounded directory subtrees. The base is the selected module directory, not its protobuf import roots or the generator file. Empty or omitted paths select all sources. Canonical portable relative paths cannot contain whitespace, dot segments, absolute paths, backslashes or globs. Global and module-local package/path filters intersect; global selectors must match across selected modules, scoped selectors must match their own module, even without plugins. Unknown selectors fail before plugins and descriptor writes. Module roots, source boundaries and required imports remain unchanged; no gitignore filtering is added. Global filters can be used without generate.modules. Generation filters are not inherited. plugins.opts.paths is a plugin output-layout option; with_imports and descriptor include_imports retain their separate meanings."} case file == v1.PolicyFile && path == "linters.extends": return []string{"Use ./ or ../ for local files, or declared-module#policy-path. Versions belong only in protobuf.mod/protobuf.lock. Local adjustments override the base; issues are not inherited. validate-config requires verified cached content and never downloads it."} case file == v1.PolicyFile && path == "issues.exclude-rules[].path": diff --git a/schemas/easyp.gen-v1.schema.json b/schemas/easyp.gen-v1.schema.json index 2f13755b..34bb3dae 100644 --- a/schemas/easyp.gen-v1.schema.json +++ b/schemas/easyp.gen-v1.schema.json @@ -631,18 +631,68 @@ "additionalProperties": false }, "modules": { + "description": "Modules selected for generation. A string selects all sources; a module object adds its own paths/packages, intersected with global filters. Omitted selects the module containing this generator. Identical repeated selections are idempotent; conflicting filters for one module are rejected.", "type": "array", "items": { - "type": "string" + "oneOf": [ + { + "type": "string", + "minLength": 1 + }, + { + "type": "object", + "properties": { + "module": { + "description": "Module identity, workspace-relative module directory, or declared dependency selected through the consumer manifest and lock.", + "type": "string", + "minLength": 1 + }, + "packages": { + "description": "Exact protobuf packages generated from this module, intersected with its paths and the project's global paths/packages. Omitted or empty means all packages. Each selector must match a resulting source in this module.", + "type": "array", + "items": { + "type": "string", + "pattern": "^[A-Za-z_][A-Za-z0-9_]*(\\.[A-Za-z_][A-Za-z0-9_]*)*$" + } + }, + "paths": { + "description": "Literal files or directory subtrees relative to the selected module directory, not its protobuf import roots or the generator file. Empty or omitted means all module sources; . selects the whole module. Paths are intersected with generate.packages and module-local filters, must be canonical portable relative names, and never change import roots or required imports. Distinct from plugins.opts.paths, which controls plugin output layout. Each selector must match a resulting source in this module.", + "type": "array", + "items": { + "type": "string", + "not": { + "pattern": "[\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]" + }, + "pattern": "^(\\.|([^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.[^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.\\.[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　])[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]*(/([^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.[^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.\\.[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　])[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]*)*)$" + } + } + }, + "additionalProperties": false, + "required": [ + "module" + ] + } + ] } }, "packages": { - "description": "Exact protobuf packages among selected modules. Empty means all source files. Imports remain available for compilation; with_imports separately controls dependency generation.", + "description": "Exact protobuf packages among selected modules, intersected with global paths and each module's paths/packages. Omitted or empty means all packages. Each selector must match a resulting source in at least one selected module. Imports remain available for compilation; with_imports separately controls dependency generation.", "type": "array", "items": { "type": "string", "pattern": "^[A-Za-z_][A-Za-z0-9_]*(\\.[A-Za-z_][A-Za-z0-9_]*)*$" } + }, + "paths": { + "description": "Literal files or directory subtrees relative to the selected module directory, not its protobuf import roots or the generator file. Empty or omitted means all module sources; . selects the whole module. Paths are intersected with generate.packages and module-local filters, must be canonical portable relative names, and never change import roots or required imports. Distinct from plugins.opts.paths, which controls plugin output layout. Each selector must match a resulting source across selected modules; module-local filters apply too. These filters can be used without generate.modules.", + "type": "array", + "items": { + "type": "string", + "not": { + "pattern": "[\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]" + }, + "pattern": "^(\\.|([^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.[^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.\\.[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　])[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]*(/([^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.[^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.\\.[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　])[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]*)*)$" + } } }, "additionalProperties": false @@ -654,7 +704,7 @@ "type": "object", "properties": { "package_prefix": { - "description": "Sets go_package without enabling managed defaults for other languages; full managed mode requires generate.managed.enabled.", + "description": "Optional Go prefix. Omitted allows inheritance; explicit empty blocks prefix inheritance. A nonempty value sets go_package without enabling managed defaults for other languages; full managed mode requires generate.managed.enabled. Migration does not create this setting.", "type": "string" } }, diff --git a/schemas/easyp.gen.schema.json b/schemas/easyp.gen.schema.json index 2f13755b..34bb3dae 100644 --- a/schemas/easyp.gen.schema.json +++ b/schemas/easyp.gen.schema.json @@ -631,18 +631,68 @@ "additionalProperties": false }, "modules": { + "description": "Modules selected for generation. A string selects all sources; a module object adds its own paths/packages, intersected with global filters. Omitted selects the module containing this generator. Identical repeated selections are idempotent; conflicting filters for one module are rejected.", "type": "array", "items": { - "type": "string" + "oneOf": [ + { + "type": "string", + "minLength": 1 + }, + { + "type": "object", + "properties": { + "module": { + "description": "Module identity, workspace-relative module directory, or declared dependency selected through the consumer manifest and lock.", + "type": "string", + "minLength": 1 + }, + "packages": { + "description": "Exact protobuf packages generated from this module, intersected with its paths and the project's global paths/packages. Omitted or empty means all packages. Each selector must match a resulting source in this module.", + "type": "array", + "items": { + "type": "string", + "pattern": "^[A-Za-z_][A-Za-z0-9_]*(\\.[A-Za-z_][A-Za-z0-9_]*)*$" + } + }, + "paths": { + "description": "Literal files or directory subtrees relative to the selected module directory, not its protobuf import roots or the generator file. Empty or omitted means all module sources; . selects the whole module. Paths are intersected with generate.packages and module-local filters, must be canonical portable relative names, and never change import roots or required imports. Distinct from plugins.opts.paths, which controls plugin output layout. Each selector must match a resulting source in this module.", + "type": "array", + "items": { + "type": "string", + "not": { + "pattern": "[\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]" + }, + "pattern": "^(\\.|([^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.[^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.\\.[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　])[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]*(/([^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.[^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.\\.[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　])[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]*)*)$" + } + } + }, + "additionalProperties": false, + "required": [ + "module" + ] + } + ] } }, "packages": { - "description": "Exact protobuf packages among selected modules. Empty means all source files. Imports remain available for compilation; with_imports separately controls dependency generation.", + "description": "Exact protobuf packages among selected modules, intersected with global paths and each module's paths/packages. Omitted or empty means all packages. Each selector must match a resulting source in at least one selected module. Imports remain available for compilation; with_imports separately controls dependency generation.", "type": "array", "items": { "type": "string", "pattern": "^[A-Za-z_][A-Za-z0-9_]*(\\.[A-Za-z_][A-Za-z0-9_]*)*$" } + }, + "paths": { + "description": "Literal files or directory subtrees relative to the selected module directory, not its protobuf import roots or the generator file. Empty or omitted means all module sources; . selects the whole module. Paths are intersected with generate.packages and module-local filters, must be canonical portable relative names, and never change import roots or required imports. Distinct from plugins.opts.paths, which controls plugin output layout. Each selector must match a resulting source across selected modules; module-local filters apply too. These filters can be used without generate.modules.", + "type": "array", + "items": { + "type": "string", + "not": { + "pattern": "[\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]" + }, + "pattern": "^(\\.|([^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.[^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.\\.[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　])[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]*(/([^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.[^/.\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]|\\.\\.[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　])[^/\\\\:*?\\[\\]\"\u003c\u003e|\\x00-\\x20\\x7f-\\x9f   - \u2028\u2029  　]*)*)$" + } } }, "additionalProperties": false @@ -654,7 +704,7 @@ "type": "object", "properties": { "package_prefix": { - "description": "Sets go_package without enabling managed defaults for other languages; full managed mode requires generate.managed.enabled.", + "description": "Optional Go prefix. Omitted allows inheritance; explicit empty blocks prefix inheritance. A nonempty value sets go_package without enabling managed defaults for other languages; full managed mode requires generate.managed.enabled. Migration does not create this setting.", "type": "string" } },