diff --git a/.agents/skills/rex-plugin-authoring/SKILL.md b/.agents/skills/rex-plugin-authoring/SKILL.md index b2cd6f7..2772170 100644 --- a/.agents/skills/rex-plugin-authoring/SKILL.md +++ b/.agents/skills/rex-plugin-authoring/SKILL.md @@ -11,7 +11,7 @@ Build for Lua 5.1 and the Rex server environment. Check the target repository's 1. Choose a unique Lua module namespace. Amber defaults `main` to the repository name, with a trailing `.rex` removed; tell users to set `main` explicitly when the entry point differs. 2. Place the entry point at `lua/
.lua` or `lua/
/init.lua`, replacing dots in `main` with directory separators. For a repository with separate Rex code, use the equivalent path under `rex/lua/`. If `rex/lua/` exists, Amber exclusively uses it and never falls back to `lua/`. -3. Return a table with a `setup(opts)` function. Amber calls it with the plugin spec's `opts` table (empty by default) on every config load, reload, and config check. Metadata is optional. Handle missing option fields, document defaults, and do not rely on setup running only once. +3. Return a table with a `setup(opts, ctx)` function. Amber calls it with the plugin spec's `opts` table (empty by default) and a context table on every config load, reload, and config check. `ctx` has `name`, `dir`, `module_dir`, `commit` (the locked commit, possibly `nil`), and `data_dir` (a state directory Amber names but does not create). Resolve paths from `ctx` rather than substituting them at install time, and tolerate `ctx` being absent in tests. Metadata is optional. Handle missing option fields, document defaults, and do not rely on setup running only once. 4. Put plugin-owned submodules beneath the main namespace, e.g. `require("example_rex.helper")` at `lua/example_rex/helper.lua`. Amber's searcher is scoped to `main` and its dotted descendants; it does not add arbitrary plugin directories to `package.path` or resolve dependencies. Lua's `package.loaded` and module names are shared, so avoid collisions. Minimal `lua/example_rex/init.lua`: @@ -35,6 +35,12 @@ return M Register actions and bindings during setup; defer operations that require a live session or client to callbacks. Config checks execute setup too, so avoid runtime, network, and filesystem side effects during setup beyond configuration. Plugins are trusted code, not sandboxed. For subprocesses use `rex.exec` with an argument array, check startup errors and nonzero exit status, and respect action deadlines; `os.execute` and `io.popen` are unavailable in Rex. +## Build steps and background processes + +If the plugin needs work after checkout, document a `build` option for the Amber spec (a shell command, a function, or a list; see the README) rather than an install script. It runs on install and update within Amber's action budget, so keep it short. Amber holds back `setup` until it has succeeded. Put build output in `.gitignore` or under `data_dir` (`AMBER_PLUGIN_DATA_DIR` for shell steps): untracked files make `amber_restore` and `amber_update` refuse the checkout. + +Amber has no service management and does not know where the `rex` binary is. For a long-running watcher, check the current Rex Lua reference, start it from `setup` with a detached `rex.exec.start`, resolve its script path from `ctx.dir`, keep state in `ctx.data_dir`, and make it exit when another instance is already running, because `setup` runs on every config load and check. State what you verified against a live Rex. + ## Document installation Give users an Amber declaration matching the actual published repository and module name, for example: diff --git a/README.md b/README.md index aa55104..bc1970c 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,7 @@ Amber installs GitHub Lua plugins on the machine running the Rex server and load - Plugin specs in Lua, in the style of lazy.nvim. - Semver versions and a lockfile for reproducible installs. +- A `build` step that runs after install or update, like lazy.nvim. - An agent skill for writing Amber plugins. ## Quick start @@ -51,7 +52,8 @@ Run with `rex do `. Add `plugin=owner/repo` to target one plugin. Withou | `amber_install` | Clones missing plugins. Uses the locked commit if one exists. Records new commits in the lockfile. | | `amber_update` | Resolves the newest commit or version, checks it out, rewrites the lock entry. Unversioned: `git pull --ff-only`. Versioned: fetch tags, check out detached. Never resets local changes. | | `amber_restore` | Checks out the locked commit (detached). Refuses dirty checkouts. Fails if the commit is unavailable. | -| `amber_status` | Shows load status, errors, path, commit, lock match, and version match. Does not fetch. | +| `amber_build` | Runs the `build` step of plugins that declare one, even if it already ran for the locked commit. | +| `amber_status` | Shows load status, errors, path, commit, lock match, build state, and version match. Does not fetch. | `amber_status` takes no `plugin` argument. Load failures also go to `rex.log`; see [Rex logs](https://www.superlogical.com/rex/docs/linux/shared-server#logs). @@ -67,6 +69,7 @@ A plugin is `"owner/repo"` or a table whose first element is that name. Unknown | `path` | none | Absolute path to a local checkout. Never installed, updated, or locked. | | `enabled` | `true` | `false` skips loading and rejects actions for it. | | `subdir` | none | Repository subdirectory to load the plugin from; requires `main`. Enables monorepo layouts. | +| `build` | none | Command, function, or list run after install or update; see [Build steps](#build-steps). Not allowed with `path`. | Options for `amber.setup` itself: @@ -84,6 +87,33 @@ For development, point `path` at a local checkout: { "rawnly/smart-splits-backend-rex", path = "/absolute/path/to/smart-splits-backend-rex", main = "smart-splits-rex" } ``` +## Build steps + +Some plugins need work after checkout, such as compiling a helper. Declare it with `build`: + +```lua +{ "alice/rex-fast", build = "make release" }, -- shell command +{ "alice/rex-fast", build = { "make deps", "make release" } }, -- steps run in order +{ + "alice/rex-fast", + build = function(ctx) + return ctx.exec({ "make", "release" }) + end, +}, +``` + +- A string runs as `sh -c ` in the plugin directory (`module_dir` for `subdir` plugins). Its environment has `AMBER_PLUGIN_NAME`, `AMBER_PLUGIN_DIR` (the checkout), `AMBER_PLUGIN_MODULE_DIR`, and `AMBER_PLUGIN_DATA_DIR`. +- A function receives `{ name, dir, module_dir, data_dir, commit, exec }`. `exec(argv, opts)` runs a command in `module_dir` within the remaining action time; `opts.env` adds to the variables above instead of replacing them. The step fails if the function raises an error, returns `false`, or returns `nil` and a message. +- `build = false` disables the step. Local `path` plugins cannot have one: Amber never runs anything for them. +- `amber_install`, `amber_update`, and `amber_restore` run the build when the locked commit differs from the one recorded as built in the lockfile (`built`). An unchanged commit is not rebuilt. `amber_build` forces it. +- A build always targets the locked commit. If the checkout has moved away from it, the build is refused; run `amber_restore` first. `amber_build` needs a lock entry, so run `amber_install` first if the lockfile was removed. +- A failed build still records the checkout in the lockfile, fails the action, and leaves the plugin unbuilt. Fix the cause and run `amber_build`. +- Keep build output out of Git's way. `amber_restore` and `amber_update` refuse checkouts with untracked or modified files, so list generated files in the plugin's `.gitignore` or write them to `data_dir`. +- Until a build has run for the locked commit, Amber does not call the plugin's `setup`; `amber_status` reports `unbuilt`. Adding `build` to an existing plugin therefore needs one `amber_build` and a reload. If the lockfile cannot be read, Amber cannot tell and does not hold plugins back. +- Steps share the roughly 23-second action budget with Git, and a bulk action shares it across all plugins. Anything longer fails with a timeout; build slow plugins one at a time with `plugin=owner/repo`, or start long-running work from the plugin itself (see [Background processes](#background-processes)). +- The build runs after the lockfile is released, so other Amber actions are not blocked by it. If the lock entry changes while it runs, the result is not recorded and `amber_build` must run again. +- Build commands are trusted code with the Rex server's privileges, like plugin Lua. Read them before installing. + ## Subdirectory plugins One repository can carry several plugins. Set `subdir` to the path each plugin loads from, relative to the repository root; `main` is required because the repository-name default cannot resolve inside a subdirectory. @@ -120,7 +150,7 @@ amber.setup({ - Tags win over branches of the same name. Branches resolve to `origin/`. - Changing `version` does not move an existing checkout. Run `amber_update`. -Amber records exact commits in the lockfile. The file is JSON and safe to commit. The lock is authoritative: `amber_install` and `amber_restore` use it until you run `amber_update`. Install never replaces an existing entry. Unrelated entries are kept and nothing is pruned. +Amber records exact commits in the lockfile, plus a `built` commit for plugins with a build step. The file is JSON and safe to commit. The lock is authoritative: `amber_install` and `amber_restore` use it until you run `amber_update`. Install never replaces an existing entry. Unrelated entries are kept and nothing is pruned. ```sh rex do amber_restore # all enabled managed plugins @@ -153,9 +183,29 @@ return M ``` - Pick a unique module name. The Lua module namespace and `package.loaded` are shared. +- `setup` also receives a second argument, `ctx`, so a plugin does not need paths filled in at install time: + + | Field | Meaning | + | --- | --- | + | `ctx.name` | Plugin identity, `owner/repo` or `owner/repo@subdir`. | + | `ctx.dir` | Checkout directory. | + | `ctx.module_dir` | Directory the module tree resolves from; differs from `dir` for `subdir` plugins. | + | `ctx.commit` | Commit recorded in the lockfile, or `nil`. | + | `ctx.data_dir` | State directory `$XDG_STATE_HOME/rex/amber//` (default `~/.local/state`). Amber names it but never creates it. | + + Plugins that ignore `ctx` keep working. Amber does not tell a plugin where the `rex` binary is. - Plugins are trusted Lua code with the Rex server's privileges. They are not sandboxed. - Config checks run `setup` too. Avoid side effects beyond configuration. +### Background processes + +Amber manages files, not processes: it has no service support and does not write launchd or systemd units. A plugin that needs a long-running watcher should start it itself from `setup`, using paths from `ctx` instead of values substituted at install time. Details to check against the current [Rex Lua reference](https://www.superlogical.com/rex/docs/reference/lua), because Amber's tests stub Rex and do not exercise them: + +- `rex.on` and `rex.every` are documented as script-only, so the watcher is a Lua script run with `rex do`, not code in `setup`. +- `rex.exec.start` with `detach = true` starts a process that outlives the config load. +- `setup` runs on every config load, reload, and `rex config check`. Make the script exit when another instance is already running (for example by labeling its client connection and checking for the label), and consider whether a check should start it at all. +- Amber does not stop such a process on update or removal; the plugin should handle restarts itself. + Install the [authoring skill](.agents/skills/rex-plugin-authoring/SKILL.md) for your agent: ```sh @@ -169,7 +219,8 @@ npx skills add rawnly/amber --skill rex-plugin-authoring - GitHub `owner/repo` only. No other Git URLs. - No dependency resolution. - No lazy loading. -- No uninstall. Remove the declaration and delete the directory yourself. +- No uninstall. Remove the declaration and delete the directory yourself. Files a build created outside the plugin directory stay too. +- No service or daemon management. ## Troubleshooting @@ -182,6 +233,7 @@ npx skills add rawnly/amber --skill rex-plugin-authoring | Lockfile error | The file is corrupt. Fix or remove it. Amber will not overwrite it. | | Checkout changed but lock write failed | Inspect the checkout and lockfile before retrying. | | `checkout differs from lockfile` warning | Run `amber_restore`. It refuses dirty checkouts. | +| Plugin reports `unbuilt` | The build has not run for the locked commit. Run `amber_build` (or fix the failing step), then `rex config reload`. | ## Contributing diff --git a/amber.lua b/amber.lua index 6116d70..b1cd558 100644 --- a/amber.lua +++ b/amber.lua @@ -11,6 +11,7 @@ local parse = assert(loadfile(directory .. "amber/spec.lua"))() local git = assert(loadfile(directory .. "amber/git.lua"))(util) local loader = assert(loadfile(directory .. "amber/loader.lua"))(util) local lock = assert(loadfile(directory .. "amber/lock.lua"))(util) +local build = assert(loadfile(directory .. "amber/build.lua"))(util) local register_action, load_plugin = loader.register_action, loader.load_plugin ---@param config? AmberConfig @@ -42,6 +43,10 @@ function M.setup(config) return end + -- Per-plugin state directories are only named here; plugins create them. + local state_home = os.getenv("XDG_STATE_HOME") or (home and home .. "/.local/state") + local state_root = state_home and state_home .. "/rex/amber" + local plugins, by_name, modules, by_dir = {}, {}, {}, {} for _, spec in ipairs(config.plugins or {}) do local ok, plugin = pcall(parse, spec, root) @@ -65,38 +70,95 @@ function M.setup(config) if not plugin.local_path then by_dir[plugin.dir] = true end + plugin.data_dir = state_root and state_root .. "/" .. plugin.slug or nil by_name[plugin.name] = plugin plugins[#plugins + 1] = plugin end end + -- Runs the Git operation and records the checkout inside one lockfile + -- transaction, then runs any pending build outside it so a slow or killed + -- build never leaves the lockfile guard behind. A second transaction records + -- the build only if the lock entry still points at the commit that was built. local function manage(plugin, deadline, operation) - return lock.transaction(lockfile, function(data) + local pending + local result, err = lock.transaction(lockfile, function(data) local entry = data.plugins[plugin.name] if operation == "restore" and not entry then return nil, "no lockfile entry for " .. plugin.name end - local status, err - if operation == "install" then - status, err = git.install(plugin, deadline, entry and entry.commit) + local status, op_err + if operation == "build" then + if plugin.local_path then + return nil, "local plugins are managed outside amber" + end + if not util.exists(plugin.dir .. "/.git/HEAD") then + return nil, "not installed" + end + if not plugin.build then + return "no build step" + end + if not entry then + return nil, "no lockfile entry; run amber_install" + end + status = "built" + elseif operation == "install" then + status, op_err = git.install(plugin, deadline, entry and entry.commit) elseif operation == "restore" then - status, err = git.restore(plugin, deadline, entry.commit) + status, op_err = git.restore(plugin, deadline, entry.commit) else - status, err = git.update(plugin, deadline) + status, op_err = git.update(plugin, deadline) end if not status then - return nil, err + return nil, op_err end - if operation == "restore" or (operation == "install" and entry) then - return status + local changed = false + if operation == "update" or (operation == "install" and not entry) then + local hash, hash_err = git.commit(plugin, deadline) + if not hash then + return nil, "checkout succeeded but lockfile was not updated: " .. tostring(hash_err) + end + -- A new commit invalidates the recorded build; an unchanged one keeps it. + local kept = entry and entry.commit == hash and entry.built or nil + entry = { commit = hash, version = plugin.version, built = kept } + data.plugins[plugin.name] = entry + changed = true end - local hash, hash_err = git.commit(plugin, deadline) - if not hash then - return nil, "checkout succeeded but lockfile was not updated: " .. tostring(hash_err) + -- The build always targets the locked commit, never whatever HEAD is. + if plugin.build and entry and (operation == "build" or entry.built ~= entry.commit) then + pending = entry.commit end - data.plugins[plugin.name] = { commit = hash, version = plugin.version } - return status, nil, true + return status, nil, changed end) + if not result or not pending then + return result, err + end + + -- The checkout and lock entry are already recorded, so a failure below + -- leaves the plugin unbuilt and amber_build can retry. + local head, head_err = git.commit(plugin, deadline) + if not head then + return nil, "cannot read checkout commit for build: " .. tostring(head_err) + end + if head ~= pending then + return nil, "checkout differs from lockfile; run amber_restore before building" + end + local built, build_err = build.run(plugin, pending, deadline) + if not built then + return nil, build_err + end + local recorded, record_err = lock.transaction(lockfile, function(data) + local entry = data.plugins[plugin.name] + if not entry or entry.commit ~= pending then + return nil, "lockfile changed during the build; run amber_build again" + end + entry.built = pending + return result, nil, true + end) + if not recorded then + return nil, "build succeeded but lockfile was not updated: " .. tostring(record_err) + end + return result end register_action("amber_install", "Install Rex Plugins", function(plugin, deadline) return manage(plugin, deadline, "install") @@ -107,6 +169,9 @@ function M.setup(config) register_action("amber_restore", "Restore Rex Plugins", function(plugin, deadline) return manage(plugin, deadline, "restore") end, by_name, plugins, true) + register_action("amber_build", "Build Rex Plugins", function(plugin, deadline) + return manage(plugin, deadline, "build") + end, by_name, plugins, true) rex.action({ name = "amber_status", title = "Rex Plugin Status", @@ -137,6 +202,10 @@ function M.setup(config) if locked and head then lock_matches = locked.commit == head end + local built + if plugin.build and locked then + built = locked.built == locked.commit + end result[plugin.name] = { enabled = plugin.enabled, path = plugin.dir, @@ -152,6 +221,7 @@ function M.setup(config) commit = head, locked_commit = locked and locked.commit or nil, lock_matches = lock_matches, + built = built, lock_error = lock_error, } end @@ -205,7 +275,28 @@ function M.setup(config) rex.log("warn", "amber: " .. plugin.name .. ": cannot check version: " .. err) end end - load_plugin(plugin) + local context = { + name = plugin.name, + dir = plugin.dir, + module_dir = plugin.module_dir, + commit = locked and locked.commit or nil, + data_dir = plugin.data_dir, + } + -- A plugin that needs a build is not set up until the build has run for + -- its locked commit, since setup typically depends on the build output. + if + plugin.enabled + and plugin.build + and lock_data + and util.exists(plugin.dir .. "/.git/HEAD") + and not (locked and locked.built == locked.commit) + then + plugin.load_status = "unbuilt" + plugin.load_error = "build has not run for the locked commit; run amber_build" + rex.log("warn", "amber: " .. plugin.name .. ": " .. plugin.load_error) + else + load_plugin(plugin, context) + end end end diff --git a/amber/build.lua b/amber/build.lua new file mode 100644 index 0000000..2c463f4 --- /dev/null +++ b/amber/build.lua @@ -0,0 +1,86 @@ +---@type AmberUtil +local util = ... + +local function remaining_budget(deadline) + if not deadline then + return nil + end + local remaining = deadline - os.time() + if remaining <= 0 then + return nil, "action time budget exhausted; run amber_build to retry" + end + return remaining +end + +-- Run every build step of a managed plugin in order, stopping at the first +-- failure. String steps run through `sh -c` with the plugin directory as the +-- working directory; function steps receive a context and fail by raising an +-- error, returning false, or returning nil and a message. All steps share the +-- action time budget, so long-running work belongs in a detached process that +-- the plugin starts itself. +---@param plugin AmberPlugin +---@param commit string +---@param deadline? number +---@return boolean? ok +---@return string? err +local function run(plugin, commit, deadline) + local base_env = { + AMBER_PLUGIN_NAME = plugin.name, + AMBER_PLUGIN_DIR = plugin.dir, + AMBER_PLUGIN_MODULE_DIR = plugin.module_dir, + AMBER_PLUGIN_DATA_DIR = plugin.data_dir, + } + ---@type AmberBuildContext + local context = { + name = plugin.name, + dir = plugin.dir, + module_dir = plugin.module_dir, + data_dir = plugin.data_dir, + commit = commit, + exec = function(argv, opts) + local timeout, budget_err = remaining_budget(deadline) + if budget_err then + return nil, budget_err + end + opts = opts or {} + -- Caller variables extend the Amber ones instead of replacing them. + local env = {} + for key, value in pairs(base_env) do + env[key] = value + end + for key, value in pairs(opts.env or {}) do + env[key] = value + end + local limit = timeout + if opts.timeout then + limit = limit and math.min(limit, opts.timeout) or opts.timeout + end + return util.run(argv, { + cwd = opts.cwd or plugin.module_dir, + env = env, + timeout = limit, + }) + end, + } + for index, step in ipairs(plugin.build or {}) do + local ok, err + if type(step) == "string" then + ok, err = context.exec({ "sh", "-c", step }) + else + local called, first, second = pcall(step, context) + if not called then + ok, err = nil, tostring(first) + elseif first == false or (first == nil and second ~= nil) then + ok, err = nil, tostring(second or "build function returned false") + else + ok = true + end + end + if not ok then + return nil, "build step " .. index .. " failed: " .. tostring(err) + end + end + return true +end + +return { run = run } diff --git a/amber/loader.lua b/amber/loader.lua index beeed73..1673b34 100644 --- a/amber/loader.lua +++ b/amber/loader.lua @@ -85,7 +85,8 @@ local function register_action(name, title, operation, plugins, ordered, all) end ---@param plugin AmberPlugin -local function load_plugin(plugin) +---@param context? AmberPluginContext +local function load_plugin(plugin, context) if not plugin.enabled then plugin.load_status = "disabled" return @@ -127,9 +128,9 @@ local function load_plugin(plugin) local module = require(plugin.main) assert( type(module) == "table" and type(module.setup) == "function", - plugin.main .. " must return a table with setup(opts)" + plugin.main .. " must return a table with setup(opts, ctx)" ) - module.setup(plugin.opts) + module.setup(plugin.opts, context) end) if not ok then plugin.load_status, plugin.load_error = "error", tostring(load_err) diff --git a/amber/lock.lua b/amber/lock.lua index 2c3367f..47c4433 100644 --- a/amber/lock.lua +++ b/amber/lock.lua @@ -42,6 +42,14 @@ function M.read(path) or not entry.commit:match("^[0-9a-f]+$") or (#entry.commit ~= 40 and #entry.commit ~= 64) or (entry.version ~= nil and type(entry.version) ~= "string") + or ( + entry.built ~= nil + and ( + type(entry.built) ~= "string" + or not entry.built:match("^[0-9a-f]+$") + or (#entry.built ~= 40 and #entry.built ~= 64) + ) + ) then return nil, "invalid lockfile entry: " .. tostring(name) end diff --git a/amber/spec.lua b/amber/spec.lua index e55a41b..7d04a5b 100644 --- a/amber/spec.lua +++ b/amber/spec.lua @@ -31,6 +31,34 @@ local function sanitize_subdir(subdir) return (subdir:gsub("/", "-"):gsub("[^%w%.%-]", "-")) end +-- Normalize the build option to a list of steps. A step is a shell command +-- string or a function; false disables the build, like lazy.nvim. +---@param build any +---@param name string +---@return (string|fun(ctx: AmberBuildContext): any)[]? +local function parse_build(build, name) + if build == nil or build == false then + return nil + end + local steps = type(build) == "table" and build or { build } + assert(#steps > 0, "plugin build must not be an empty list: " .. name) + for key in pairs(steps) do + assert( + type(key) == "number" and key >= 1 and key <= #steps and key % 1 == 0, + "plugin build must be a command, a function, or a list of them: " .. name + ) + end + local normalized = {} + for i, step in ipairs(steps) do + assert( + type(step) == "function" or (type(step) == "string" and step:match("%S")), + "plugin build step " .. i .. " must be a command string or a function: " .. name + ) + normalized[i] = step + end + return normalized +end + ---@param spec string|AmberPluginSpec ---@param root string ---@return AmberPlugin @@ -42,7 +70,7 @@ local function parse(spec, root) type(spec) == "table" and type(spec[1]) == "string", "plugin must be an owner/repository string or table" ) - local options = { "opts", "enabled", "path", "main", "version", "subdir" } + local options = { "opts", "enabled", "path", "main", "version", "subdir", "build" } local allowed = { [1] = true, opts = true, @@ -51,6 +79,7 @@ local function parse(spec, root) main = true, version = true, subdir = true, + build = true, } for key in pairs(spec) do if not allowed[key] then @@ -113,6 +142,11 @@ local function parse(spec, root) spec.path == nil or (type(spec.path) == "string" and spec.path:sub(1, 1) == "/"), "plugin path must be absolute: " .. spec[1] ) + assert( + not spec.path or not spec.build, + "build is not supported for local plugins: " .. spec[1] .. " (Amber never runs them)" + ) + local build = parse_build(spec.build, spec[1]) assert( not subdir or type(spec.main) == "string", "main is required for subdir plugins: " @@ -131,10 +165,11 @@ local function parse(spec, root) -- subdirectory so lockfile, status, and per-plugin action targets stay -- unique while module resolution looks inside that subdirectory. local name = subdir and (spec[1] .. "@" .. subdir) or spec[1] - local dir = spec.path - or (root .. "/" .. owner .. "/" .. repo .. (subdir and "--" .. sanitize_subdir(subdir) or "")) + local slug = owner .. "/" .. repo .. (subdir and "--" .. sanitize_subdir(subdir) or "") + local dir = spec.path or (root .. "/" .. slug) return { name = name, + slug = slug, subdir = subdir, url = "https://github.com/" .. spec[1] .. ".git", dir = dir, @@ -147,6 +182,7 @@ local function parse(spec, root) local_path = spec.path ~= nil, enabled = spec.enabled ~= false, version = spec.version, + build = build, } end diff --git a/docs/architecture.md b/docs/architecture.md index e60307f..bcddd3d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,6 +1,6 @@ # Architecture -Amber is a Lua 5.1 module loaded by the Rex server's configuration. It turns plugin declarations into local paths, loads their Rex-side Lua entry points, and registers actions for installing, updating, restoring, and inspecting GitHub repositories. The entry point is `amber.lua`; `amber/spec.lua` validates specs, `amber/git.lua` handles Git operations, `amber/lock.lua` manages the lockfile, `amber/loader.lua` registers actions and loads plugins, and `amber/util.lua` contains shared command and filesystem helpers. +Amber is a Lua 5.1 module loaded by the Rex server's configuration. It turns plugin declarations into local paths, loads their Rex-side Lua entry points, and registers actions for installing, updating, restoring, and inspecting GitHub repositories. The entry point is `amber.lua`; `amber/spec.lua` validates specs, `amber/git.lua` handles Git operations, `amber/lock.lua` manages the lockfile, `amber/loader.lua` registers actions and loads plugins, `amber/build.lua` runs build steps, and `amber/util.lua` contains shared command and filesystem helpers. ## Execution model @@ -8,13 +8,14 @@ Amber is a Lua 5.1 module loaded by the Rex server's configuration. It turns plu Rex config load / check / reload └─ amber.setup(config) ├─ validate and normalize plugin specifications - ├─ register amber_install, amber_update, amber_restore, and amber_status actions + ├─ register amber_install, amber_update, amber_restore, amber_build, and amber_status actions └─ load enabled entry points, in declaration order - └─ require(main).setup(opts) + └─ require(main).setup(opts, ctx) Later: rex do amber_install [plugin=owner/repo] rex do amber_update [plugin=owner/repo] rex do amber_restore [plugin=owner/repo] + rex do amber_build [plugin=owner/repo] rex do amber_status └─ Rex invokes the corresponding registered action └─ Amber runs Git and filesystem commands via rex.exec @@ -36,10 +37,11 @@ Rex configuration runs on the server machine. Amber, local plugin paths, Git, an | `main` | Lua module name; defaults to the repository name. | | `version` | Git ref or supported semver constraint for managed checkouts; incompatible with `path`. | | `subdir` | Repository subdirectory the plugin loads from; requires `main`, which stops defaulting to the repository name. | +| `build` | Shell command, function, or list of them run after install or update; rejected with `path`. `false` disables it. | All plugins, including local ones, have a unique identity. Plain specs keep the `owner/repo` identity; `subdir` specs identify as `owner/repo@subdir`, which keys the lockfile, status, and per-plugin action targets, and cannot collide with a plain name. Amber validates names, module names, and subdir paths, logs and skips invalid specifications, duplicate identities, enabled duplicate main module names, and duplicate install locations across all non-local specs, enabled or disabled (including a sanitized subdir install name that collides with a plain repository), and derives a GitHub HTTPS URL for each repository. For managed plugins, the default root is `$XDG_DATA_HOME/rex/plugins` or `~/.local/share/rex/plugins`; plain checkouts live at `//`, and a subdir plugin clones to `//--`; both stay one path segment. `root` overrides that location. A local `path` bypasses this layout and is never installed or updated by Amber. -Amber keeps both an ordered list for loading and a name-indexed table for action lookup. These are built anew each time `setup` runs. Plugin modules must return a table containing `setup(opts)`; version and author fields in that module are not required. `version` in the manager's specification controls the checkout, not plugin metadata. +Amber keeps both an ordered list for loading and a name-indexed table for action lookup. These are built anew each time `setup` runs. Plugin modules must return a table containing `setup(opts, ctx)`; `ctx` carries `name`, `dir`, `module_dir`, the locked `commit`, and a `data_dir` that Amber names but does not create, and a plugin may ignore it; version and author fields in that module are not required. `version` in the manager's specification controls the checkout, not plugin metadata. ## Loading and errors @@ -64,13 +66,23 @@ For updates, an unversioned checkout uses `git pull --ff-only`. A versioned chec `rex.exec` receives an argument vector rather than a shell command. Amber checks both process startup errors and exit status; network Git operations set `GIT_TERMINAL_PROMPT=0`. Individual Git operations have a 20-second timeout, but Rex actions also have their own deadline, so a multi-command operation can still be interrupted by Rex. +## Build steps + +A managed plugin may declare `build`, normalized by `amber/spec.lua` into a list of steps. `manage` in `amber.lua` works in three phases. First, one lockfile transaction runs the Git operation of `amber_install`, `amber_update`, or `amber_restore` (none for `amber_build`), writes the lock entry, and notes a pending build when the entry's `built` differs from its `commit`, or always for `amber_build`. `amber_build` requires an existing lock entry. Second, with the lockfile guard released, it checks that `HEAD` equals the locked commit and runs the steps through `amber/build.lua`; a drifted checkout is refused with a pointer to `amber_restore`. Third, a second transaction sets `built` to the commit, but only if the entry still locks that commit; otherwise the action fails and the build must be repeated. + +String steps run as `sh -c` in `module_dir` through `rex.exec` with `AMBER_PLUGIN_NAME`, `AMBER_PLUGIN_DIR`, `AMBER_PLUGIN_MODULE_DIR`, and `AMBER_PLUGIN_DATA_DIR`; function steps get a context whose `exec` applies the remaining action budget and merges caller variables over those. The first failing step stops the build. Running builds outside the transaction keeps a slow or interrupted build from leaving the `.lock` guard behind. + +A failed build leaves the recorded checkout in place with `built` unset, so `amber_build` can retry. An update to a different commit drops `built`; an update that lands on the same commit keeps it. Build output inside the checkout must be ignored by Git, since restore and detached updates refuse untracked files. + +At config load, an enabled plugin with a build step whose checkout exists and whose lock entry lacks `built == commit` is not set up: its load status is `unbuilt` and a warning is logged. If the lockfile cannot be read, this check is skipped. Config load never runs a build. Builds are trusted code and share the 23-second action budget with Git, so long-running work must be started by the plugin itself. + ## Version resolution Without `version`, Git's default branch is used. Exact tags take precedence over remote branch names, and branch names resolve to `origin/`. `amber_status` compares the checked-out commit with the locally resolved version without fetching. `^x.y.z` and `~x.y.z` select the highest matching stable `x.y.z` or `vx.y.z` tag among locally available tags, applying caret or tilde bounds. Updates fetch tags before resolving again. Prerelease tags and broader lazy.nvim semver expressions are not supported. The lockfile pins resolved commits across installs; a range can resolve to a newer tag on an explicit update. ## Lockfile -The default `amber-lock.json` is next to Rex's configuration, with an absolute-path override in `setup`. It stores a format version and a map of plugin identities to exact Git commits and their declared version. Configuration loads only read it. Manual install uses an existing lock entry or records the resulting checkout when no entry exists; update replaces the entry after a successful checkout. Restore uses locked commits and refuses dirty working trees. Local and disabled plugins are skipped in bulk operations, and unrelated lock entries are retained. +The default `amber-lock.json` is next to Rex's configuration, with an absolute-path override in `setup`. It stores a format version and a map of plugin identities to exact Git commits, their declared version, and, for plugins with a build step, the commit last built (`built`). Configuration loads only read it. Manual install uses an existing lock entry or records the resulting checkout when no entry exists; update replaces the entry after a successful checkout. Restore uses locked commits and refuses dirty working trees. Local and disabled plugins are skipped in bulk operations, and unrelated lock entries are retained. Operations serialize lockfile read-modify-write with a sibling `.lock` directory. Writes use a unique temporary sibling and rename. A crash can leave the guard behind; operators must verify no action is running before removing it. A failed lock write after Git changed a checkout leaves that checkout changed and returns an error; the filesystem and Git update are not a single transaction. Restore checks out detached commits, and can fail if a locked commit is no longer available. @@ -82,4 +94,4 @@ Operations serialize lockfile read-modify-write with a sibling `.lock` directory A spec can name a repository subdirectory it loads from. Identity carries the subdirectory (`owner/repo@subdir`) and becomes the lockfile key, the status key, and the per-plugin action target. Managed installs clone the whole repository into `//--`, where `--` separates the repository from the subdirectory and `/` and other symbols map to `-`; two entries of one repository therefore track different commits. Modules resolve from the subdirectory's own `rex/lua` or `lua/` tree, so each entry is self-contained. `main` is required for subdir specs. Local `path` checkouts combine with `subdir` as a module offset. Duplicate install locations are reported at setup for every non-local spec, enabled or disabled, so a disabled checkout cannot satisfy an enabled plugin. -Current non-goals include dependency resolution, lazy loading, uninstall, arbitrary remote URLs, and Windows support. These are not implied by the current module contract. +Current non-goals include dependency resolution, lazy loading, uninstall, service or daemon management, arbitrary remote URLs, and Windows support. These are not implied by the current module contract. diff --git a/scripts/test.sh b/scripts/test.sh index d09c0df..462bf32 100644 --- a/scripts/test.sh +++ b/scripts/test.sh @@ -10,7 +10,8 @@ mkdir -p "$fixture/alice/example/.git" "$fixture/alice/example/lua/example" "$fi "$fixture/elly/utility--panes-bells/.git" "$fixture/elly/utility--panes-bells/panes/bells/lua/bells" \ "$fixture/local-mono/panes/tools/lua/tools" \ "$fixture/temporary2/.git" "$fixture/temporary2/panes/bells/lua/bells" \ - "$fixture/temporary3/.git" "$fixture/temporary4/.git" "$fixture/elly/claimed--gone/.git" + "$fixture/temporary3/.git" "$fixture/temporary4/.git" "$fixture/elly/claimed--gone/.git" \ + "$fixture/dave/built/.git" "$fixture/dave/built/lua/built" "$fixture/dave/plain/.git" "$fixture/dave/plain/lua/plain" printf 'ref: refs/heads/main\n' >"$fixture/alice/example/.git/HEAD" printf 'ref: refs/heads/main\n' >"$fixture/carol/shared/.git/HEAD" printf 'ref: refs/heads/main\n' >"$fixture/temporary/.git/HEAD" @@ -32,4 +33,8 @@ printf '%s\n' 'return { setup = function() _G.subdir_message = "bells" end }' >" printf '%s\n' 'return { setup = function() _G.subdir_message = "local" end }' >"$fixture/local-mono/panes/tools/lua/tools/init.lua" printf '%s\n' 'return { setup = function() end }' >"$fixture/temporary2/panes/bells/lua/bells/init.lua" printf '%s\n' 'return { setup = function() _G.shared_source = "nvim" end }' >"$fixture/carol/shared/lua/shared/init.lua" +printf 'ref: refs/heads/main\n' >"$fixture/dave/built/.git/HEAD" +printf 'ref: refs/heads/main\n' >"$fixture/dave/plain/.git/HEAD" +printf '%s\n' 'return { setup = function(_, ctx) _G.built_ctx = ctx end }' >"$fixture/dave/built/lua/built/init.lua" +printf '%s\n' 'return { setup = function(_, ctx) _G.plain_ctx = ctx end }' >"$fixture/dave/plain/lua/plain/init.lua" lua tests/test.lua "$fixture" diff --git a/tests/test.lua b/tests/test.lua index 335829d..b757ca2 100644 --- a/tests/test.lua +++ b/tests/test.lua @@ -21,6 +21,8 @@ local function test(name, fn) end package.loaded.example, package.loaded.shared, package.loaded.keys, package.loaded.tools = nil, nil, nil, nil + package.loaded.built, package.loaded.plain = nil, nil + _G.built_ctx, _G.plain_ctx = nil, nil _G.plugin_message = nil _G.subdir_message = nil rex = { @@ -66,6 +68,15 @@ local function test(name, fn) code = 0, } end + if argv[1] == "sh" and argv[2] == "-c" then + local failed = behavior.build_fail + return { + ok = not failed, + stdout = "", + stderr = failed and "build exploded" or "", + code = failed and 1 or 0, + } + end if argv[1] == "git" and argv[2] == "clone" then if behavior.clone_success then return { ok = true, stdout = "", stderr = "", code = 0 } @@ -600,6 +611,230 @@ test("updates a clean detached checkout after restore", function(amber, actions, assert(checkout and checkout.argv[4] == "refs/remotes/origin/main") end) +local function count_calls(calls, command) + local total = 0 + for _, call in ipairs(calls) do + if call.argv[1] == command then + total = total + 1 + end + end + return total +end + +test("passes a runtime context to setup", function(amber, actions) + amber.setup({ root = root, plugins = { "dave/plain" } }) + local context = assert(_G.plain_ctx, "setup must receive a context") + assert(context.name == "dave/plain" and context.dir == root .. "/dave/plain") + assert(context.module_dir == context.dir and context.commit == nil) + assert(context.data_dir:match("/rex/amber/dave/plain$")) + actions.amber_install.run({}, { plugin = "dave/plain" }) + amber.setup({ root = root, plugins = { "dave/plain" } }) + assert(_G.plain_ctx.commit == string.rep("a", 40)) +end) + +test("builds on install and gates setup until built", function(amber, actions, logs, calls) + local spec = { "dave/built", build = "make" } + amber.setup({ root = root, plugins = { spec } }) + assert(_G.built_ctx == nil, "setup must wait for the build") + assert(logs[#logs][2]:match("run amber_build")) + assert(actions.amber_status.run({}, {})["dave/built"].loaded == "unbuilt") + assert(actions.amber_install.run({}, { plugin = "dave/built" }).status == "already installed") + local build = assert(called(calls, "sh", "-c")) + assert(build.argv[3] == "make" and build.opts.cwd == root .. "/dave/built") + assert(build.opts.env.AMBER_PLUGIN_NAME == "dave/built") + assert(build.opts.timeout and build.opts.timeout <= 23) + amber.setup({ root = root, plugins = { spec } }) + assert(_G.built_ctx and _G.built_ctx.commit == string.rep("a", 40)) + assert(actions.amber_status.run({}, {})["dave/built"].built == true) + actions.amber_install.run({}, { plugin = "dave/built" }) + assert(count_calls(calls, "sh") == 1, "an unchanged commit must not rebuild") +end) + +test("a failed build is retryable", function(amber, actions, _, calls, behavior) + amber.setup({ root = root, plugins = { { "dave/built", build = { "make", "make install" } } } }) + behavior.build_fail = true + local ok, err = pcall(actions.amber_install.run, {}, { plugin = "dave/built" }) + assert(not ok and tostring(err):match("build step 1 failed: build exploded"), tostring(err)) + assert(count_calls(calls, "sh") == 1, "later steps must not run after a failure") + local entry = behavior.lock_data.plugins["dave/built"] + assert(entry.commit == string.rep("a", 40) and entry.built == nil) + assert(actions.amber_status.run({}, {})["dave/built"].built == false) + behavior.build_fail = false + assert(actions.amber_build.run({}, { plugin = "dave/built" }).status == "built") + assert(count_calls(calls, "sh") == 3) + assert(behavior.lock_data.plugins["dave/built"].built == string.rep("a", 40)) +end) + +test("amber_build rebuilds and skips plugins without one", function(amber, actions, _, calls, b) + local behavior = b + amber.setup({ root = root, plugins = { { "dave/built", build = "make" }, "dave/plain" } }) + actions.amber_install.run({}, {}) + assert(count_calls(calls, "sh") == 1) + local result = actions.amber_build.run({}, {}) + assert(result.plugins["dave/built"].status == "built") + assert(result.plugins["dave/plain"].status == "no build step") + assert(count_calls(calls, "sh") == 2, "amber_build must rebuild even when already built") + assert(behavior.lock_data.plugins["dave/built"].built == string.rep("a", 40)) + amber.setup({ root = root, plugins = { { "bob/missing", build = "make" } } }) + local ok, err = pcall(actions.amber_build.run, {}, { plugin = "bob/missing" }) + assert(not ok and tostring(err):match("not installed")) +end) + +test("a new commit invalidates the recorded build", function(amber, actions, _, calls, behavior) + amber.setup({ root = root, plugins = { { "dave/built", build = "make" } } }) + actions.amber_install.run({}, { plugin = "dave/built" }) + actions.amber_update.run({}, { plugin = "dave/built" }) + assert(count_calls(calls, "sh") == 1, "an update to the same commit keeps the build") + behavior.head_hash = string.rep("b", 40) + actions.amber_update.run({}, { plugin = "dave/built" }) + assert(count_calls(calls, "sh") == 2) + local entry = behavior.lock_data.plugins["dave/built"] + assert(entry.commit == string.rep("b", 40) and entry.built == string.rep("b", 40)) +end) + +test("runs function build steps and reports failures", function(amber, actions, _, _, behavior) + local seen + local mode = "ok" + amber.setup({ + root = root, + plugins = { + { + "dave/built", + build = function(ctx) + seen = ctx + if mode == "false" then + return false + elseif mode == "message" then + return nil, "no compiler" + elseif mode == "raise" then + error("boom", 0) + end + return ctx.exec({ "sh", "-c", "true" }) + end, + }, + }, + }) + assert(actions.amber_install.run({}, { plugin = "dave/built" }).status == "already installed") + assert(seen.name == "dave/built" and seen.commit == string.rep("a", 40)) + assert(seen.dir == root .. "/dave/built" and seen.module_dir == seen.dir) + assert(behavior.lock_data.plugins["dave/built"].built == string.rep("a", 40)) + for failure, expected in pairs({ + ["false"] = "returned false", + message = "no compiler", + raise = "boom", + }) do + mode = failure + local ok, err = pcall(actions.amber_build.run, {}, { plugin = "dave/built" }) + assert(not ok and tostring(err):match(expected), failure .. ": " .. tostring(err)) + end +end) + +test("validates build specifications", function(amber, _, logs) + for _, build in ipairs({ 7, true, {}, { 1 }, { "" }, { make = "make" }, " " }) do + amber.setup({ + root = root, + install = "manual", + ---@diagnostic disable-next-line: assign-type-mismatch + plugins = { { "dave/built", build = build } }, + }) + assert(logs[#logs][2]:match("invalid plugin specification"), "accepted " .. tostring(build)) + end + amber.setup({ + root = root, + install = "manual", + plugins = { { "dave/built", path = root .. "/dave/built", build = "make" } }, + }) + assert(logs[#logs][2]:match("build is not supported for local plugins")) + local before = #logs + amber.setup({ root = root, plugins = { { "dave/plain", build = false } } }) + assert(#logs == before and _G.plain_ctx, "build = false must disable the build") +end) + +test("rejects malformed built entries in the lockfile", function() + local path = root .. "/test-lock.json" + local file = assert(io.open(path, "w")) + file:write("{}") + file:close() + local lock = assert(loadfile("amber/lock.lua"))(dofile("amber/util.lua")) + for _, built in ipairs({ "xyz", 7 }) do + ---@diagnostic disable-next-line: duplicate-set-field + rex.json.decode = function() + return { + format = 1, + plugins = { ["a/b"] = { commit = string.rep("a", 40), built = built } }, + } + end + local data, err = lock.read(path) + assert(not data and err:match("invalid lockfile entry"), tostring(built)) + end + ---@diagnostic disable-next-line: duplicate-set-field + rex.json.decode = function() + return { + format = 1, + plugins = { ["a/b"] = { commit = string.rep("a", 40), built = string.rep("a", 40) } }, + } + end + assert(lock.read(path)) +end) + +test("amber_build requires a lock entry", function(amber, actions, _, calls) + amber.setup({ root = root, plugins = { { "dave/built", build = "make" } } }) + local ok, err = pcall(actions.amber_build.run, {}, { plugin = "dave/built" }) + assert(not ok and tostring(err):match("no lockfile entry"), tostring(err)) + assert(count_calls(calls, "sh") == 0, "nothing may be built without a lock entry") +end) + +test("does not build a drifted checkout", function(amber, actions, _, calls, behavior) + amber.setup({ root = root, plugins = { { "dave/built", build = "make" } } }) + actions.amber_install.run({}, { plugin = "dave/built" }) + assert(count_calls(calls, "sh") == 1) + behavior.head_hash = string.rep("b", 40) + local ok, err = pcall(actions.amber_build.run, {}, { plugin = "dave/built" }) + assert(not ok and tostring(err):match("checkout differs from lockfile"), tostring(err)) + assert(count_calls(calls, "sh") == 1, "a drifted checkout must not be built") + assert(behavior.lock_data.plugins["dave/built"].built == string.rep("a", 40)) +end) + +test("builds outside the lockfile guard", function(amber, actions, _, calls, behavior) + local guard = root .. "/test-lock.json.lock" + local function held() + local depth = 0 + for _, call in ipairs(calls) do + if call.argv[3] == guard then + depth = depth + (call.argv[1] == "mkdir" and 1 or -1) + end + end + return depth + end + local during + local move_lock = false + amber.setup({ + root = root, + plugins = { + { + "dave/built", + build = function(ctx) + during = held() + if move_lock then + behavior.lock_data.plugins["dave/built"].commit = string.rep("c", 40) + end + return ctx.exec({ "sh", "-c", "true" }, { env = { EXTRA = "1" } }) + end, + }, + }, + }) + actions.amber_install.run({}, { plugin = "dave/built" }) + assert(during == 0, "the lockfile guard must be released before building") + local build = assert(called(calls, "sh", "-c")) + assert(build.opts.env.EXTRA == "1" and build.opts.env.AMBER_PLUGIN_NAME == "dave/built") + assert(build.opts.env.AMBER_PLUGIN_DIR == root .. "/dave/built") + assert(build.opts.env.AMBER_PLUGIN_MODULE_DIR == root .. "/dave/built") + move_lock = true + local ok, err = pcall(actions.amber_build.run, {}, { plugin = "dave/built" }) + assert(not ok and tostring(err):match("lockfile changed during the build"), tostring(err)) + assert(behavior.lock_data.plugins["dave/built"].built == string.rep("a", 40)) +end) + test("rejects expired operation budgets", function() local git = assert(loadfile("amber/git.lua"))(dofile("amber/util.lua")) local status, err = git.install({ local_path = false }, os.time() - 1) diff --git a/types/amber.lua b/types/amber.lua index 5a4bbe0..04a69ad 100644 --- a/types/amber.lua +++ b/types/amber.lua @@ -8,6 +8,22 @@ ---@field path? string Absolute path to an externally managed checkout. ---@field main? string Lua module name; defaults to the repository name. ---@field version? string Git ref or supported semver constraint. +---@field build? string|(string|fun(ctx: AmberBuildContext): any)[]|fun(ctx: AmberBuildContext): any|false Command, function, or list run after install or update; see the README. + +---@class AmberPluginContext Second argument of a plugin's setup function. +---@field name string Plugin identity, `owner/repo` or `owner/repo@subdir`. +---@field dir string Checkout directory. +---@field module_dir string Directory the module tree resolves from; differs from dir for subdir plugins. +---@field commit? string Commit recorded in the lockfile, if any. +---@field data_dir? string Per-plugin state directory under `$XDG_STATE_HOME/rex/amber`; Amber does not create it. + +---@class AmberBuildContext +---@field name string +---@field dir string +---@field module_dir string +---@field data_dir? string Per-plugin state directory; Amber does not create it. +---@field commit string Locked commit being built; the checkout is verified to match it. +---@field exec fun(argv: string[], opts?: RexExecOptions): RexExecResult?, string? Runs a command in module_dir within the remaining action budget; opts.env extends the AMBER_PLUGIN_* variables. ---@class AmberConfig ---@field root? string Absolute directory for managed checkouts. @@ -17,6 +33,7 @@ ---@class AmberPlugin ---@field name string +---@field slug string Relative install path: `owner/repo`, plus `--subdir` for subdir plugins. ---@field url string ---@field dir string ---@field subdir? string @@ -27,6 +44,8 @@ ---@field local_path boolean ---@field enabled boolean ---@field version? string +---@field build? (string|fun(ctx: AmberBuildContext): any)[] +---@field data_dir? string Per-plugin state directory, set by setup. ---@field load_status? string ---@field load_error? string