From accaa38e49596f2bf07ec903210e11b4fa6cbabb Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 14 Nov 2026 10:03:37 +0100 Subject: [PATCH 1/3] Add build steps and a runtime context for plugins Plugins that need work after checkout, or that must resolve paths at runtime, had to ship their own installer. Add the lazy.nvim-style pieces Amber was missing: - build option (command, function, or list) run after install, update, or restore when the checkout commit differs from the lockfile's built commit - amber_build action to force a build, and a built state in amber_status - lockfile entries may carry a built commit; a failed build still records the checkout and can be retried - setup is held back (status unbuilt) until the build has run for the locked commit - setup(opts, ctx) with name, dir, module_dir, commit, and data_dir - docs on starting background processes from the plugin; Amber still has no service management Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01HYR5Nyv85TQGevuubCSZYd --- .agents/skills/rex-plugin-authoring/SKILL.md | 8 +- README.md | 55 +++++- amber.lua | 88 ++++++++-- amber/build.lua | 72 ++++++++ amber/loader.lua | 7 +- amber/lock.lua | 8 + amber/spec.lua | 42 ++++- docs/architecture.md | 22 ++- scripts/test.sh | 7 +- tests/test.lua | 175 +++++++++++++++++++ types/amber.lua | 17 ++ 11 files changed, 474 insertions(+), 27 deletions(-) create mode 100644 amber/build.lua diff --git a/.agents/skills/rex-plugin-authoring/SKILL.md b/.agents/skills/rex-plugin-authoring/SKILL.md index b2cd6f7..69abeb2 100644 --- a/.agents/skills/rex-plugin-authoring/SKILL.md +++ b/.agents/skills/rex-plugin-authoring/SKILL.md @@ -11,7 +11,7 @@ Build for Lua 5.1 and the Rex server environment. Check the target repository's 1. Choose a unique Lua module namespace. Amber defaults `main` to the repository name, with a trailing `.rex` removed; tell users to set `main` explicitly when the entry point differs. 2. Place the entry point at `lua/
.lua` or `lua/
/init.lua`, replacing dots in `main` with directory separators. For a repository with separate Rex code, use the equivalent path under `rex/lua/`. If `rex/lua/` exists, Amber exclusively uses it and never falls back to `lua/`. -3. Return a table with a `setup(opts)` function. Amber calls it with the plugin spec's `opts` table (empty by default) on every config load, reload, and config check. Metadata is optional. Handle missing option fields, document defaults, and do not rely on setup running only once. +3. Return a table with a `setup(opts, ctx)` function. Amber calls it with the plugin spec's `opts` table (empty by default) and a context table on every config load, reload, and config check. `ctx` has `name`, `dir`, `module_dir`, `commit` (the locked commit, possibly `nil`), and `data_dir` (a state directory Amber names but does not create). Resolve paths from `ctx` rather than substituting them at install time, and tolerate `ctx` being absent in tests. Metadata is optional. Handle missing option fields, document defaults, and do not rely on setup running only once. 4. Put plugin-owned submodules beneath the main namespace, e.g. `require("example_rex.helper")` at `lua/example_rex/helper.lua`. Amber's searcher is scoped to `main` and its dotted descendants; it does not add arbitrary plugin directories to `package.path` or resolve dependencies. Lua's `package.loaded` and module names are shared, so avoid collisions. Minimal `lua/example_rex/init.lua`: @@ -35,6 +35,12 @@ return M Register actions and bindings during setup; defer operations that require a live session or client to callbacks. Config checks execute setup too, so avoid runtime, network, and filesystem side effects during setup beyond configuration. Plugins are trusted code, not sandboxed. For subprocesses use `rex.exec` with an argument array, check startup errors and nonzero exit status, and respect action deadlines; `os.execute` and `io.popen` are unavailable in Rex. +## Build steps and background processes + +If the plugin needs work after checkout, document a `build` option for the Amber spec (a shell command, a function, or a list; see the README) rather than an install script. It runs on install and update within Amber's action budget, so keep it short. Amber holds back `setup` until it has succeeded. + +Amber has no service management and does not know where the `rex` binary is. For a long-running watcher, check the current Rex Lua reference, start it from `setup` with a detached `rex.exec.start`, resolve its script path from `ctx.dir`, keep state in `ctx.data_dir`, and make it exit when another instance is already running, because `setup` runs on every config load and check. State what you verified against a live Rex. + ## Document installation Give users an Amber declaration matching the actual published repository and module name, for example: diff --git a/README.md b/README.md index aa55104..d950831 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,7 @@ Amber installs GitHub Lua plugins on the machine running the Rex server and load - Plugin specs in Lua, in the style of lazy.nvim. - Semver versions and a lockfile for reproducible installs. +- A `build` step that runs after install or update, like lazy.nvim. - An agent skill for writing Amber plugins. ## Quick start @@ -51,7 +52,8 @@ Run with `rex do `. Add `plugin=owner/repo` to target one plugin. Withou | `amber_install` | Clones missing plugins. Uses the locked commit if one exists. Records new commits in the lockfile. | | `amber_update` | Resolves the newest commit or version, checks it out, rewrites the lock entry. Unversioned: `git pull --ff-only`. Versioned: fetch tags, check out detached. Never resets local changes. | | `amber_restore` | Checks out the locked commit (detached). Refuses dirty checkouts. Fails if the commit is unavailable. | -| `amber_status` | Shows load status, errors, path, commit, lock match, and version match. Does not fetch. | +| `amber_build` | Runs the `build` step of plugins that declare one, even if it already ran for the locked commit. | +| `amber_status` | Shows load status, errors, path, commit, lock match, build state, and version match. Does not fetch. | `amber_status` takes no `plugin` argument. Load failures also go to `rex.log`; see [Rex logs](https://www.superlogical.com/rex/docs/linux/shared-server#logs). @@ -67,6 +69,7 @@ A plugin is `"owner/repo"` or a table whose first element is that name. Unknown | `path` | none | Absolute path to a local checkout. Never installed, updated, or locked. | | `enabled` | `true` | `false` skips loading and rejects actions for it. | | `subdir` | none | Repository subdirectory to load the plugin from; requires `main`. Enables monorepo layouts. | +| `build` | none | Command, function, or list run after install or update; see [Build steps](#build-steps). Not allowed with `path`. | Options for `amber.setup` itself: @@ -84,6 +87,30 @@ For development, point `path` at a local checkout: { "rawnly/smart-splits-backend-rex", path = "/absolute/path/to/smart-splits-backend-rex", main = "smart-splits-rex" } ``` +## Build steps + +Some plugins need work after checkout, such as compiling a helper. Declare it with `build`: + +```lua +{ "alice/rex-fast", build = "make release" }, -- shell command +{ "alice/rex-fast", build = { "make deps", "make release" } }, -- steps run in order +{ + "alice/rex-fast", + build = function(ctx) + return ctx.exec({ "make", "release" }) + end, +}, +``` + +- A string runs as `sh -c ` in the plugin directory (`module_dir` for `subdir` plugins). `AMBER_PLUGIN_NAME` and `AMBER_PLUGIN_DIR` are set in its environment. +- A function receives `{ name, dir, module_dir, commit, exec }`. `exec(argv, opts)` runs a command in `module_dir` within the remaining action time. The step fails if the function raises an error, returns `false`, or returns `nil` and a message. +- `build = false` disables the step. Local `path` plugins cannot have one: Amber never runs anything for them. +- `amber_install`, `amber_update`, and `amber_restore` run the build when the checked-out commit differs from the one recorded as built in the lockfile (`built`). An unchanged commit is not rebuilt. `amber_build` forces it. +- A failed build still records the checkout in the lockfile, fails the action, and leaves the plugin unbuilt. Fix the cause and run `amber_build`. +- Until a build has run for the locked commit, Amber does not call the plugin's `setup`; `amber_status` reports `unbuilt`. Adding `build` to an existing plugin therefore needs one `amber_build` and a reload. If the lockfile cannot be read, Amber cannot tell and does not hold plugins back. +- Steps share the roughly 23-second action budget with Git. Anything longer fails with a timeout; start long-running work from the plugin itself instead (see [Background processes](#background-processes)). +- Build commands are trusted code with the Rex server's privileges, like plugin Lua. Read them before installing. + ## Subdirectory plugins One repository can carry several plugins. Set `subdir` to the path each plugin loads from, relative to the repository root; `main` is required because the repository-name default cannot resolve inside a subdirectory. @@ -120,7 +147,7 @@ amber.setup({ - Tags win over branches of the same name. Branches resolve to `origin/`. - Changing `version` does not move an existing checkout. Run `amber_update`. -Amber records exact commits in the lockfile. The file is JSON and safe to commit. The lock is authoritative: `amber_install` and `amber_restore` use it until you run `amber_update`. Install never replaces an existing entry. Unrelated entries are kept and nothing is pruned. +Amber records exact commits in the lockfile, plus a `built` commit for plugins with a build step. The file is JSON and safe to commit. The lock is authoritative: `amber_install` and `amber_restore` use it until you run `amber_update`. Install never replaces an existing entry. Unrelated entries are kept and nothing is pruned. ```sh rex do amber_restore # all enabled managed plugins @@ -153,9 +180,29 @@ return M ``` - Pick a unique module name. The Lua module namespace and `package.loaded` are shared. +- `setup` also receives a second argument, `ctx`, so a plugin does not need paths filled in at install time: + + | Field | Meaning | + | --- | --- | + | `ctx.name` | Plugin identity, `owner/repo` or `owner/repo@subdir`. | + | `ctx.dir` | Checkout directory. | + | `ctx.module_dir` | Directory the module tree resolves from; differs from `dir` for `subdir` plugins. | + | `ctx.commit` | Commit recorded in the lockfile, or `nil`. | + | `ctx.data_dir` | State directory `$XDG_STATE_HOME/rex/amber//` (default `~/.local/state`). Amber names it but never creates it. | + + Plugins that ignore `ctx` keep working. Amber does not tell a plugin where the `rex` binary is. - Plugins are trusted Lua code with the Rex server's privileges. They are not sandboxed. - Config checks run `setup` too. Avoid side effects beyond configuration. +### Background processes + +Amber manages files, not processes: it has no service support and does not write launchd or systemd units. A plugin that needs a long-running watcher should start it itself from `setup`, using paths from `ctx` instead of values substituted at install time. Details to check against the current [Rex Lua reference](https://www.superlogical.com/rex/docs/reference/lua), because Amber's tests stub Rex and do not exercise them: + +- `rex.on` and `rex.every` are documented as script-only, so the watcher is a Lua script run with `rex do`, not code in `setup`. +- `rex.exec.start` with `detach = true` starts a process that outlives the config load. +- `setup` runs on every config load, reload, and `rex config check`. Make the script exit when another instance is already running (for example by labeling its client connection and checking for the label), and consider whether a check should start it at all. +- Amber does not stop such a process on update or removal; the plugin should handle restarts itself. + Install the [authoring skill](.agents/skills/rex-plugin-authoring/SKILL.md) for your agent: ```sh @@ -169,7 +216,8 @@ npx skills add rawnly/amber --skill rex-plugin-authoring - GitHub `owner/repo` only. No other Git URLs. - No dependency resolution. - No lazy loading. -- No uninstall. Remove the declaration and delete the directory yourself. +- No uninstall. Remove the declaration and delete the directory yourself. Files a build created outside the plugin directory stay too. +- No service or daemon management. ## Troubleshooting @@ -182,6 +230,7 @@ npx skills add rawnly/amber --skill rex-plugin-authoring | Lockfile error | The file is corrupt. Fix or remove it. Amber will not overwrite it. | | Checkout changed but lock write failed | Inspect the checkout and lockfile before retrying. | | `checkout differs from lockfile` warning | Run `amber_restore`. It refuses dirty checkouts. | +| Plugin reports `unbuilt` | The build has not run for the locked commit. Run `amber_build` (or fix the failing step), then `rex config reload`. | ## Contributing diff --git a/amber.lua b/amber.lua index 6116d70..47a9b61 100644 --- a/amber.lua +++ b/amber.lua @@ -11,6 +11,7 @@ local parse = assert(loadfile(directory .. "amber/spec.lua"))() local git = assert(loadfile(directory .. "amber/git.lua"))(util) local loader = assert(loadfile(directory .. "amber/loader.lua"))(util) local lock = assert(loadfile(directory .. "amber/lock.lua"))(util) +local build = assert(loadfile(directory .. "amber/build.lua"))(util) local register_action, load_plugin = loader.register_action, loader.load_plugin ---@param config? AmberConfig @@ -42,6 +43,10 @@ function M.setup(config) return end + -- Per-plugin state directories are only named here; plugins create them. + local state_home = os.getenv("XDG_STATE_HOME") or (home and home .. "/.local/state") + local state_root = state_home and state_home .. "/rex/amber" + local plugins, by_name, modules, by_dir = {}, {}, {}, {} for _, spec in ipairs(config.plugins or {}) do local ok, plugin = pcall(parse, spec, root) @@ -71,13 +76,22 @@ function M.setup(config) end local function manage(plugin, deadline, operation) - return lock.transaction(lockfile, function(data) + local build_err + local result, err = lock.transaction(lockfile, function(data) local entry = data.plugins[plugin.name] if operation == "restore" and not entry then return nil, "no lockfile entry for " .. plugin.name end local status, err - if operation == "install" then + if operation == "build" then + if plugin.local_path then + return nil, "local plugins are managed outside amber" + end + if not util.exists(plugin.dir .. "/.git/HEAD") then + return nil, "not installed" + end + status = plugin.build and "built" or "no build step" + elseif operation == "install" then status, err = git.install(plugin, deadline, entry and entry.commit) elseif operation == "restore" then status, err = git.restore(plugin, deadline, entry.commit) @@ -87,16 +101,41 @@ function M.setup(config) if not status then return nil, err end - if operation == "restore" or (operation == "install" and entry) then - return status + local changed = false + local hash, hash_err + if operation == "update" or (operation == "install" and not entry) then + hash, hash_err = git.commit(plugin, deadline) + if not hash then + return nil, "checkout succeeded but lockfile was not updated: " .. tostring(hash_err) + end + -- A new commit invalidates the recorded build; an unchanged one keeps it. + local kept = entry and entry.commit == hash and entry.built or nil + entry = { commit = hash, version = plugin.version, built = kept } + data.plugins[plugin.name] = entry + changed = true end - local hash, hash_err = git.commit(plugin, deadline) - if not hash then - return nil, "checkout succeeded but lockfile was not updated: " .. tostring(hash_err) + if plugin.build and entry then + hash, hash_err = hash or git.commit(plugin, deadline) + if not hash then + build_err = "cannot read checkout commit for build: " .. tostring(hash_err) + elseif operation == "build" or entry.built ~= hash then + local built, built_err = build.run(plugin, hash, deadline) + if built then + entry.built = hash + changed = true + else + build_err = built_err + end + end end - data.plugins[plugin.name] = { commit = hash, version = plugin.version } - return status, nil, true + return status, nil, changed end) + -- The checkout and lock entry are recorded even when the build fails, so + -- the failure is reported after the transaction and amber_build can retry. + if result and build_err then + return nil, build_err + end + return result, err end register_action("amber_install", "Install Rex Plugins", function(plugin, deadline) return manage(plugin, deadline, "install") @@ -107,6 +146,9 @@ function M.setup(config) register_action("amber_restore", "Restore Rex Plugins", function(plugin, deadline) return manage(plugin, deadline, "restore") end, by_name, plugins, true) + register_action("amber_build", "Build Rex Plugins", function(plugin, deadline) + return manage(plugin, deadline, "build") + end, by_name, plugins, true) rex.action({ name = "amber_status", title = "Rex Plugin Status", @@ -137,6 +179,10 @@ function M.setup(config) if locked and head then lock_matches = locked.commit == head end + local built + if plugin.build and locked then + built = locked.built == locked.commit + end result[plugin.name] = { enabled = plugin.enabled, path = plugin.dir, @@ -152,6 +198,7 @@ function M.setup(config) commit = head, locked_commit = locked and locked.commit or nil, lock_matches = lock_matches, + built = built, lock_error = lock_error, } end @@ -205,7 +252,28 @@ function M.setup(config) rex.log("warn", "amber: " .. plugin.name .. ": cannot check version: " .. err) end end - load_plugin(plugin) + local context = { + name = plugin.name, + dir = plugin.dir, + module_dir = plugin.module_dir, + commit = locked and locked.commit or nil, + data_dir = state_root and state_root .. "/" .. plugin.slug or nil, + } + -- A plugin that needs a build is not set up until the build has run for + -- its locked commit, since setup typically depends on the build output. + if + plugin.enabled + and plugin.build + and lock_data + and util.exists(plugin.dir .. "/.git/HEAD") + and not (locked and locked.built == locked.commit) + then + plugin.load_status = "unbuilt" + plugin.load_error = "build has not run for the locked commit; run amber_build" + rex.log("warn", "amber: " .. plugin.name .. ": " .. plugin.load_error) + else + load_plugin(plugin, context) + end end end diff --git a/amber/build.lua b/amber/build.lua new file mode 100644 index 0000000..0044858 --- /dev/null +++ b/amber/build.lua @@ -0,0 +1,72 @@ +---@type AmberUtil +local util = ... + +local function remaining_budget(deadline) + if not deadline then + return nil + end + local remaining = deadline - os.time() + if remaining <= 0 then + return nil, "action time budget exhausted; run amber_build to retry" + end + return remaining +end + +-- Run every build step of a managed plugin in order, stopping at the first +-- failure. String steps run through `sh -c` with the plugin directory as the +-- working directory; function steps receive a context and fail by raising an +-- error, returning false, or returning nil and a message. All steps share the +-- action time budget, so long-running work belongs in a detached process that +-- the plugin starts itself. +---@param plugin AmberPlugin +---@param commit string +---@param deadline? number +---@return boolean? ok +---@return string? err +local function run(plugin, commit, deadline) + local env = { AMBER_PLUGIN_NAME = plugin.name, AMBER_PLUGIN_DIR = plugin.module_dir } + ---@type AmberBuildContext + local context = { + name = plugin.name, + dir = plugin.dir, + module_dir = plugin.module_dir, + commit = commit, + exec = function(argv, opts) + local timeout, budget_err = remaining_budget(deadline) + if budget_err then + return nil, budget_err + end + opts = opts or {} + local limit = timeout + if opts.timeout then + limit = limit and math.min(limit, opts.timeout) or opts.timeout + end + return util.run(argv, { + cwd = opts.cwd or plugin.module_dir, + env = opts.env or env, + timeout = limit, + }) + end, + } + for index, step in ipairs(plugin.build or {}) do + local ok, err + if type(step) == "string" then + ok, err = context.exec({ "sh", "-c", step }) + else + local called, first, second = pcall(step, context) + if not called then + ok, err = nil, tostring(first) + elseif first == false or (first == nil and second ~= nil) then + ok, err = nil, tostring(second or "build function returned false") + else + ok = true + end + end + if not ok then + return nil, "build step " .. index .. " failed: " .. tostring(err) + end + end + return true +end + +return { run = run } diff --git a/amber/loader.lua b/amber/loader.lua index beeed73..1673b34 100644 --- a/amber/loader.lua +++ b/amber/loader.lua @@ -85,7 +85,8 @@ local function register_action(name, title, operation, plugins, ordered, all) end ---@param plugin AmberPlugin -local function load_plugin(plugin) +---@param context? AmberPluginContext +local function load_plugin(plugin, context) if not plugin.enabled then plugin.load_status = "disabled" return @@ -127,9 +128,9 @@ local function load_plugin(plugin) local module = require(plugin.main) assert( type(module) == "table" and type(module.setup) == "function", - plugin.main .. " must return a table with setup(opts)" + plugin.main .. " must return a table with setup(opts, ctx)" ) - module.setup(plugin.opts) + module.setup(plugin.opts, context) end) if not ok then plugin.load_status, plugin.load_error = "error", tostring(load_err) diff --git a/amber/lock.lua b/amber/lock.lua index 2c3367f..47c4433 100644 --- a/amber/lock.lua +++ b/amber/lock.lua @@ -42,6 +42,14 @@ function M.read(path) or not entry.commit:match("^[0-9a-f]+$") or (#entry.commit ~= 40 and #entry.commit ~= 64) or (entry.version ~= nil and type(entry.version) ~= "string") + or ( + entry.built ~= nil + and ( + type(entry.built) ~= "string" + or not entry.built:match("^[0-9a-f]+$") + or (#entry.built ~= 40 and #entry.built ~= 64) + ) + ) then return nil, "invalid lockfile entry: " .. tostring(name) end diff --git a/amber/spec.lua b/amber/spec.lua index e55a41b..7d04a5b 100644 --- a/amber/spec.lua +++ b/amber/spec.lua @@ -31,6 +31,34 @@ local function sanitize_subdir(subdir) return (subdir:gsub("/", "-"):gsub("[^%w%.%-]", "-")) end +-- Normalize the build option to a list of steps. A step is a shell command +-- string or a function; false disables the build, like lazy.nvim. +---@param build any +---@param name string +---@return (string|fun(ctx: AmberBuildContext): any)[]? +local function parse_build(build, name) + if build == nil or build == false then + return nil + end + local steps = type(build) == "table" and build or { build } + assert(#steps > 0, "plugin build must not be an empty list: " .. name) + for key in pairs(steps) do + assert( + type(key) == "number" and key >= 1 and key <= #steps and key % 1 == 0, + "plugin build must be a command, a function, or a list of them: " .. name + ) + end + local normalized = {} + for i, step in ipairs(steps) do + assert( + type(step) == "function" or (type(step) == "string" and step:match("%S")), + "plugin build step " .. i .. " must be a command string or a function: " .. name + ) + normalized[i] = step + end + return normalized +end + ---@param spec string|AmberPluginSpec ---@param root string ---@return AmberPlugin @@ -42,7 +70,7 @@ local function parse(spec, root) type(spec) == "table" and type(spec[1]) == "string", "plugin must be an owner/repository string or table" ) - local options = { "opts", "enabled", "path", "main", "version", "subdir" } + local options = { "opts", "enabled", "path", "main", "version", "subdir", "build" } local allowed = { [1] = true, opts = true, @@ -51,6 +79,7 @@ local function parse(spec, root) main = true, version = true, subdir = true, + build = true, } for key in pairs(spec) do if not allowed[key] then @@ -113,6 +142,11 @@ local function parse(spec, root) spec.path == nil or (type(spec.path) == "string" and spec.path:sub(1, 1) == "/"), "plugin path must be absolute: " .. spec[1] ) + assert( + not spec.path or not spec.build, + "build is not supported for local plugins: " .. spec[1] .. " (Amber never runs them)" + ) + local build = parse_build(spec.build, spec[1]) assert( not subdir or type(spec.main) == "string", "main is required for subdir plugins: " @@ -131,10 +165,11 @@ local function parse(spec, root) -- subdirectory so lockfile, status, and per-plugin action targets stay -- unique while module resolution looks inside that subdirectory. local name = subdir and (spec[1] .. "@" .. subdir) or spec[1] - local dir = spec.path - or (root .. "/" .. owner .. "/" .. repo .. (subdir and "--" .. sanitize_subdir(subdir) or "")) + local slug = owner .. "/" .. repo .. (subdir and "--" .. sanitize_subdir(subdir) or "") + local dir = spec.path or (root .. "/" .. slug) return { name = name, + slug = slug, subdir = subdir, url = "https://github.com/" .. spec[1] .. ".git", dir = dir, @@ -147,6 +182,7 @@ local function parse(spec, root) local_path = spec.path ~= nil, enabled = spec.enabled ~= false, version = spec.version, + build = build, } end diff --git a/docs/architecture.md b/docs/architecture.md index e60307f..b0810df 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,6 +1,6 @@ # Architecture -Amber is a Lua 5.1 module loaded by the Rex server's configuration. It turns plugin declarations into local paths, loads their Rex-side Lua entry points, and registers actions for installing, updating, restoring, and inspecting GitHub repositories. The entry point is `amber.lua`; `amber/spec.lua` validates specs, `amber/git.lua` handles Git operations, `amber/lock.lua` manages the lockfile, `amber/loader.lua` registers actions and loads plugins, and `amber/util.lua` contains shared command and filesystem helpers. +Amber is a Lua 5.1 module loaded by the Rex server's configuration. It turns plugin declarations into local paths, loads their Rex-side Lua entry points, and registers actions for installing, updating, restoring, and inspecting GitHub repositories. The entry point is `amber.lua`; `amber/spec.lua` validates specs, `amber/git.lua` handles Git operations, `amber/lock.lua` manages the lockfile, `amber/loader.lua` registers actions and loads plugins, `amber/build.lua` runs build steps, and `amber/util.lua` contains shared command and filesystem helpers. ## Execution model @@ -8,13 +8,14 @@ Amber is a Lua 5.1 module loaded by the Rex server's configuration. It turns plu Rex config load / check / reload └─ amber.setup(config) ├─ validate and normalize plugin specifications - ├─ register amber_install, amber_update, amber_restore, and amber_status actions + ├─ register amber_install, amber_update, amber_restore, amber_build, and amber_status actions └─ load enabled entry points, in declaration order - └─ require(main).setup(opts) + └─ require(main).setup(opts, ctx) Later: rex do amber_install [plugin=owner/repo] rex do amber_update [plugin=owner/repo] rex do amber_restore [plugin=owner/repo] + rex do amber_build [plugin=owner/repo] rex do amber_status └─ Rex invokes the corresponding registered action └─ Amber runs Git and filesystem commands via rex.exec @@ -36,10 +37,11 @@ Rex configuration runs on the server machine. Amber, local plugin paths, Git, an | `main` | Lua module name; defaults to the repository name. | | `version` | Git ref or supported semver constraint for managed checkouts; incompatible with `path`. | | `subdir` | Repository subdirectory the plugin loads from; requires `main`, which stops defaulting to the repository name. | +| `build` | Shell command, function, or list of them run after install or update; rejected with `path`. `false` disables it. | All plugins, including local ones, have a unique identity. Plain specs keep the `owner/repo` identity; `subdir` specs identify as `owner/repo@subdir`, which keys the lockfile, status, and per-plugin action targets, and cannot collide with a plain name. Amber validates names, module names, and subdir paths, logs and skips invalid specifications, duplicate identities, enabled duplicate main module names, and duplicate install locations across all non-local specs, enabled or disabled (including a sanitized subdir install name that collides with a plain repository), and derives a GitHub HTTPS URL for each repository. For managed plugins, the default root is `$XDG_DATA_HOME/rex/plugins` or `~/.local/share/rex/plugins`; plain checkouts live at `//`, and a subdir plugin clones to `//--`; both stay one path segment. `root` overrides that location. A local `path` bypasses this layout and is never installed or updated by Amber. -Amber keeps both an ordered list for loading and a name-indexed table for action lookup. These are built anew each time `setup` runs. Plugin modules must return a table containing `setup(opts)`; version and author fields in that module are not required. `version` in the manager's specification controls the checkout, not plugin metadata. +Amber keeps both an ordered list for loading and a name-indexed table for action lookup. These are built anew each time `setup` runs. Plugin modules must return a table containing `setup(opts, ctx)`; `ctx` carries `name`, `dir`, `module_dir`, the locked `commit`, and a `data_dir` that Amber names but does not create, and a plugin may ignore it; version and author fields in that module are not required. `version` in the manager's specification controls the checkout, not plugin metadata. ## Loading and errors @@ -64,13 +66,21 @@ For updates, an unversioned checkout uses `git pull --ff-only`. A versioned chec `rex.exec` receives an argument vector rather than a shell command. Amber checks both process startup errors and exit status; network Git operations set `GIT_TERMINAL_PROMPT=0`. Individual Git operations have a 20-second timeout, but Rex actions also have their own deadline, so a multi-command operation can still be interrupted by Rex. +## Build steps + +A managed plugin may declare `build`, normalized by `amber/spec.lua` into a list of steps. After the Git operation of `amber_install`, `amber_update`, or `amber_restore`, `manage` in `amber.lua` reads the checkout commit and runs the steps through `amber/build.lua` when the lock entry's `built` differs from it; `amber_build` skips the Git step and always runs them. String steps run as `sh -c` in `module_dir` through `rex.exec`; function steps get a context whose `exec` applies the remaining action budget. The first failing step stops the build. + +The lock entry is written inside the same lockfile transaction, with `built` set to the commit only on success. A failed build still records the checkout, then the action fails, so a retry through `amber_build` has a lock entry to work with. An update to a different commit drops `built`; an update that lands on the same commit keeps it. + +At config load, an enabled plugin with a build step whose checkout exists and whose lock entry lacks `built == commit` is not set up: its load status is `unbuilt` and a warning is logged. If the lockfile cannot be read, this check is skipped. Config load never runs a build. Builds are trusted code and share the 23-second action budget with Git, so long-running work must be started by the plugin itself. + ## Version resolution Without `version`, Git's default branch is used. Exact tags take precedence over remote branch names, and branch names resolve to `origin/`. `amber_status` compares the checked-out commit with the locally resolved version without fetching. `^x.y.z` and `~x.y.z` select the highest matching stable `x.y.z` or `vx.y.z` tag among locally available tags, applying caret or tilde bounds. Updates fetch tags before resolving again. Prerelease tags and broader lazy.nvim semver expressions are not supported. The lockfile pins resolved commits across installs; a range can resolve to a newer tag on an explicit update. ## Lockfile -The default `amber-lock.json` is next to Rex's configuration, with an absolute-path override in `setup`. It stores a format version and a map of plugin identities to exact Git commits and their declared version. Configuration loads only read it. Manual install uses an existing lock entry or records the resulting checkout when no entry exists; update replaces the entry after a successful checkout. Restore uses locked commits and refuses dirty working trees. Local and disabled plugins are skipped in bulk operations, and unrelated lock entries are retained. +The default `amber-lock.json` is next to Rex's configuration, with an absolute-path override in `setup`. It stores a format version and a map of plugin identities to exact Git commits, their declared version, and, for plugins with a build step, the commit last built (`built`). Configuration loads only read it. Manual install uses an existing lock entry or records the resulting checkout when no entry exists; update replaces the entry after a successful checkout. Restore uses locked commits and refuses dirty working trees. Local and disabled plugins are skipped in bulk operations, and unrelated lock entries are retained. Operations serialize lockfile read-modify-write with a sibling `.lock` directory. Writes use a unique temporary sibling and rename. A crash can leave the guard behind; operators must verify no action is running before removing it. A failed lock write after Git changed a checkout leaves that checkout changed and returns an error; the filesystem and Git update are not a single transaction. Restore checks out detached commits, and can fail if a locked commit is no longer available. @@ -82,4 +92,4 @@ Operations serialize lockfile read-modify-write with a sibling `.lock` directory A spec can name a repository subdirectory it loads from. Identity carries the subdirectory (`owner/repo@subdir`) and becomes the lockfile key, the status key, and the per-plugin action target. Managed installs clone the whole repository into `//--`, where `--` separates the repository from the subdirectory and `/` and other symbols map to `-`; two entries of one repository therefore track different commits. Modules resolve from the subdirectory's own `rex/lua` or `lua/` tree, so each entry is self-contained. `main` is required for subdir specs. Local `path` checkouts combine with `subdir` as a module offset. Duplicate install locations are reported at setup for every non-local spec, enabled or disabled, so a disabled checkout cannot satisfy an enabled plugin. -Current non-goals include dependency resolution, lazy loading, uninstall, arbitrary remote URLs, and Windows support. These are not implied by the current module contract. +Current non-goals include dependency resolution, lazy loading, uninstall, service or daemon management, arbitrary remote URLs, and Windows support. These are not implied by the current module contract. diff --git a/scripts/test.sh b/scripts/test.sh index d09c0df..462bf32 100644 --- a/scripts/test.sh +++ b/scripts/test.sh @@ -10,7 +10,8 @@ mkdir -p "$fixture/alice/example/.git" "$fixture/alice/example/lua/example" "$fi "$fixture/elly/utility--panes-bells/.git" "$fixture/elly/utility--panes-bells/panes/bells/lua/bells" \ "$fixture/local-mono/panes/tools/lua/tools" \ "$fixture/temporary2/.git" "$fixture/temporary2/panes/bells/lua/bells" \ - "$fixture/temporary3/.git" "$fixture/temporary4/.git" "$fixture/elly/claimed--gone/.git" + "$fixture/temporary3/.git" "$fixture/temporary4/.git" "$fixture/elly/claimed--gone/.git" \ + "$fixture/dave/built/.git" "$fixture/dave/built/lua/built" "$fixture/dave/plain/.git" "$fixture/dave/plain/lua/plain" printf 'ref: refs/heads/main\n' >"$fixture/alice/example/.git/HEAD" printf 'ref: refs/heads/main\n' >"$fixture/carol/shared/.git/HEAD" printf 'ref: refs/heads/main\n' >"$fixture/temporary/.git/HEAD" @@ -32,4 +33,8 @@ printf '%s\n' 'return { setup = function() _G.subdir_message = "bells" end }' >" printf '%s\n' 'return { setup = function() _G.subdir_message = "local" end }' >"$fixture/local-mono/panes/tools/lua/tools/init.lua" printf '%s\n' 'return { setup = function() end }' >"$fixture/temporary2/panes/bells/lua/bells/init.lua" printf '%s\n' 'return { setup = function() _G.shared_source = "nvim" end }' >"$fixture/carol/shared/lua/shared/init.lua" +printf 'ref: refs/heads/main\n' >"$fixture/dave/built/.git/HEAD" +printf 'ref: refs/heads/main\n' >"$fixture/dave/plain/.git/HEAD" +printf '%s\n' 'return { setup = function(_, ctx) _G.built_ctx = ctx end }' >"$fixture/dave/built/lua/built/init.lua" +printf '%s\n' 'return { setup = function(_, ctx) _G.plain_ctx = ctx end }' >"$fixture/dave/plain/lua/plain/init.lua" lua tests/test.lua "$fixture" diff --git a/tests/test.lua b/tests/test.lua index 335829d..a2a54eb 100644 --- a/tests/test.lua +++ b/tests/test.lua @@ -21,6 +21,8 @@ local function test(name, fn) end package.loaded.example, package.loaded.shared, package.loaded.keys, package.loaded.tools = nil, nil, nil, nil + package.loaded.built, package.loaded.plain = nil, nil + _G.built_ctx, _G.plain_ctx = nil, nil _G.plugin_message = nil _G.subdir_message = nil rex = { @@ -66,6 +68,15 @@ local function test(name, fn) code = 0, } end + if argv[1] == "sh" and argv[2] == "-c" then + local failed = behavior.build_fail + return { + ok = not failed, + stdout = "", + stderr = failed and "build exploded" or "", + code = failed and 1 or 0, + } + end if argv[1] == "git" and argv[2] == "clone" then if behavior.clone_success then return { ok = true, stdout = "", stderr = "", code = 0 } @@ -600,6 +611,170 @@ test("updates a clean detached checkout after restore", function(amber, actions, assert(checkout and checkout.argv[4] == "refs/remotes/origin/main") end) +local function count_calls(calls, command) + local total = 0 + for _, call in ipairs(calls) do + if call.argv[1] == command then + total = total + 1 + end + end + return total +end + +test("passes a runtime context to setup", function(amber, actions) + amber.setup({ root = root, plugins = { "dave/plain" } }) + local context = assert(_G.plain_ctx, "setup must receive a context") + assert(context.name == "dave/plain" and context.dir == root .. "/dave/plain") + assert(context.module_dir == context.dir and context.commit == nil) + assert(context.data_dir:match("/rex/amber/dave/plain$")) + actions.amber_install.run({}, { plugin = "dave/plain" }) + amber.setup({ root = root, plugins = { "dave/plain" } }) + assert(_G.plain_ctx.commit == string.rep("a", 40)) +end) + +test("builds on install and gates setup until built", function(amber, actions, logs, calls) + local spec = { "dave/built", build = "make" } + amber.setup({ root = root, plugins = { spec } }) + assert(_G.built_ctx == nil, "setup must wait for the build") + assert(logs[#logs][2]:match("run amber_build")) + assert(actions.amber_status.run({}, {})["dave/built"].loaded == "unbuilt") + assert(actions.amber_install.run({}, { plugin = "dave/built" }).status == "already installed") + local build = assert(called(calls, "sh", "-c")) + assert(build.argv[3] == "make" and build.opts.cwd == root .. "/dave/built") + assert(build.opts.env.AMBER_PLUGIN_NAME == "dave/built") + assert(build.opts.timeout and build.opts.timeout <= 23) + amber.setup({ root = root, plugins = { spec } }) + assert(_G.built_ctx and _G.built_ctx.commit == string.rep("a", 40)) + assert(actions.amber_status.run({}, {})["dave/built"].built == true) + actions.amber_install.run({}, { plugin = "dave/built" }) + assert(count_calls(calls, "sh") == 1, "an unchanged commit must not rebuild") +end) + +test("a failed build is retryable", function(amber, actions, _, calls, behavior) + amber.setup({ root = root, plugins = { { "dave/built", build = { "make", "make install" } } } }) + behavior.build_fail = true + local ok, err = pcall(actions.amber_install.run, {}, { plugin = "dave/built" }) + assert(not ok and tostring(err):match("build step 1 failed: build exploded"), tostring(err)) + assert(count_calls(calls, "sh") == 1, "later steps must not run after a failure") + local entry = behavior.lock_data.plugins["dave/built"] + assert(entry.commit == string.rep("a", 40) and entry.built == nil) + assert(actions.amber_status.run({}, {})["dave/built"].built == false) + behavior.build_fail = false + assert(actions.amber_build.run({}, { plugin = "dave/built" }).status == "built") + assert(count_calls(calls, "sh") == 3) + assert(behavior.lock_data.plugins["dave/built"].built == string.rep("a", 40)) +end) + +test("amber_build rebuilds and skips plugins without one", function(amber, actions, _, calls, b) + local behavior = b + amber.setup({ root = root, plugins = { { "dave/built", build = "make" }, "dave/plain" } }) + actions.amber_install.run({}, {}) + assert(count_calls(calls, "sh") == 1) + local result = actions.amber_build.run({}, {}) + assert(result.plugins["dave/built"].status == "built") + assert(result.plugins["dave/plain"].status == "no build step") + assert(count_calls(calls, "sh") == 2, "amber_build must rebuild even when already built") + assert(behavior.lock_data.plugins["dave/built"].built == string.rep("a", 40)) + amber.setup({ root = root, plugins = { { "bob/missing", build = "make" } } }) + local ok, err = pcall(actions.amber_build.run, {}, { plugin = "bob/missing" }) + assert(not ok and tostring(err):match("not installed")) +end) + +test("a new commit invalidates the recorded build", function(amber, actions, _, calls, behavior) + amber.setup({ root = root, plugins = { { "dave/built", build = "make" } } }) + actions.amber_install.run({}, { plugin = "dave/built" }) + actions.amber_update.run({}, { plugin = "dave/built" }) + assert(count_calls(calls, "sh") == 1, "an update to the same commit keeps the build") + behavior.head_hash = string.rep("b", 40) + actions.amber_update.run({}, { plugin = "dave/built" }) + assert(count_calls(calls, "sh") == 2) + local entry = behavior.lock_data.plugins["dave/built"] + assert(entry.commit == string.rep("b", 40) and entry.built == string.rep("b", 40)) +end) + +test("runs function build steps and reports failures", function(amber, actions, _, _, behavior) + local seen + local mode = "ok" + amber.setup({ + root = root, + plugins = { + { + "dave/built", + build = function(ctx) + seen = ctx + if mode == "false" then + return false + elseif mode == "message" then + return nil, "no compiler" + elseif mode == "raise" then + error("boom", 0) + end + return ctx.exec({ "sh", "-c", "true" }) + end, + }, + }, + }) + assert(actions.amber_install.run({}, { plugin = "dave/built" }).status == "already installed") + assert(seen.name == "dave/built" and seen.commit == string.rep("a", 40)) + assert(seen.dir == root .. "/dave/built" and seen.module_dir == seen.dir) + assert(behavior.lock_data.plugins["dave/built"].built == string.rep("a", 40)) + for failure, expected in pairs({ + ["false"] = "returned false", + message = "no compiler", + raise = "boom", + }) do + mode = failure + local ok, err = pcall(actions.amber_build.run, {}, { plugin = "dave/built" }) + assert(not ok and tostring(err):match(expected), failure .. ": " .. tostring(err)) + end +end) + +test("validates build specifications", function(amber, _, logs) + for _, build in ipairs({ 7, true, {}, { 1 }, { "" }, { make = "make" }, " " }) do + amber.setup({ + root = root, + install = "manual", + ---@diagnostic disable-next-line: assign-type-mismatch + plugins = { { "dave/built", build = build } }, + }) + assert(logs[#logs][2]:match("invalid plugin specification"), "accepted " .. tostring(build)) + end + amber.setup({ + root = root, + install = "manual", + plugins = { { "dave/built", path = root .. "/dave/built", build = "make" } }, + }) + assert(logs[#logs][2]:match("build is not supported for local plugins")) + local before = #logs + amber.setup({ root = root, plugins = { { "dave/plain", build = false } } }) + assert(#logs == before and _G.plain_ctx, "build = false must disable the build") +end) + +test("rejects malformed built entries in the lockfile", function() + local path = root .. "/test-lock.json" + local file = assert(io.open(path, "w")) + file:write("{}") + file:close() + local lock = assert(loadfile("amber/lock.lua"))(dofile("amber/util.lua")) + for _, built in ipairs({ "xyz", 7 }) do + rex.json.decode = function() + return { + format = 1, + plugins = { ["a/b"] = { commit = string.rep("a", 40), built = built } }, + } + end + local data, err = lock.read(path) + assert(not data and err:match("invalid lockfile entry"), tostring(built)) + end + rex.json.decode = function() + return { + format = 1, + plugins = { ["a/b"] = { commit = string.rep("a", 40), built = string.rep("a", 40) } }, + } + end + assert(lock.read(path)) +end) + test("rejects expired operation budgets", function() local git = assert(loadfile("amber/git.lua"))(dofile("amber/util.lua")) local status, err = git.install({ local_path = false }, os.time() - 1) diff --git a/types/amber.lua b/types/amber.lua index 5a4bbe0..650e238 100644 --- a/types/amber.lua +++ b/types/amber.lua @@ -8,6 +8,21 @@ ---@field path? string Absolute path to an externally managed checkout. ---@field main? string Lua module name; defaults to the repository name. ---@field version? string Git ref or supported semver constraint. +---@field build? string|(string|fun(ctx: AmberBuildContext): any)[]|fun(ctx: AmberBuildContext): any|false Command, function, or list run after install or update; see the README. + +---@class AmberPluginContext Second argument of a plugin's setup function. +---@field name string Plugin identity, `owner/repo` or `owner/repo@subdir`. +---@field dir string Checkout directory. +---@field module_dir string Directory the module tree resolves from; differs from dir for subdir plugins. +---@field commit? string Commit recorded in the lockfile, if any. +---@field data_dir? string Per-plugin state directory under `$XDG_STATE_HOME/rex/amber`; Amber does not create it. + +---@class AmberBuildContext +---@field name string +---@field dir string +---@field module_dir string +---@field commit string Checked-out commit being built. +---@field exec fun(argv: string[], opts?: RexExecOptions): RexExecResult?, string? Runs a command in module_dir within the remaining action budget. ---@class AmberConfig ---@field root? string Absolute directory for managed checkouts. @@ -17,6 +32,7 @@ ---@class AmberPlugin ---@field name string +---@field slug string Relative install path: `owner/repo`, plus `--subdir` for subdir plugins. ---@field url string ---@field dir string ---@field subdir? string @@ -27,6 +43,7 @@ ---@field local_path boolean ---@field enabled boolean ---@field version? string +---@field build? (string|fun(ctx: AmberBuildContext): any)[] ---@field load_status? string ---@field load_error? string From 04ca22ffea4f3391712c8abcc9adac92e8e96eb7 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 14 Nov 2026 10:06:05 +0100 Subject: [PATCH 2/3] Fix Lua language server warnings in build changes Split an unbalanced assignment in amber.lua and annotate the test stubs that replace rex.json.decode, which the types check reported. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01HYR5Nyv85TQGevuubCSZYd --- amber.lua | 4 +++- tests/test.lua | 2 ++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/amber.lua b/amber.lua index 47a9b61..f71af1f 100644 --- a/amber.lua +++ b/amber.lua @@ -115,7 +115,9 @@ function M.setup(config) changed = true end if plugin.build and entry then - hash, hash_err = hash or git.commit(plugin, deadline) + if not hash then + hash, hash_err = git.commit(plugin, deadline) + end if not hash then build_err = "cannot read checkout commit for build: " .. tostring(hash_err) elseif operation == "build" or entry.built ~= hash then diff --git a/tests/test.lua b/tests/test.lua index a2a54eb..2bc3684 100644 --- a/tests/test.lua +++ b/tests/test.lua @@ -757,6 +757,7 @@ test("rejects malformed built entries in the lockfile", function() file:close() local lock = assert(loadfile("amber/lock.lua"))(dofile("amber/util.lua")) for _, built in ipairs({ "xyz", 7 }) do + ---@diagnostic disable-next-line: duplicate-set-field rex.json.decode = function() return { format = 1, @@ -766,6 +767,7 @@ test("rejects malformed built entries in the lockfile", function() local data, err = lock.read(path) assert(not data and err:match("invalid lockfile entry"), tostring(built)) end + ---@diagnostic disable-next-line: duplicate-set-field rex.json.decode = function() return { format = 1, From 5e84c506fc8df6f45d4631fdf93a0af526bdc829 Mon Sep 17 00:00:00 2001 From: Federico <16429579+rawnly@users.noreply.github.com> Date: Sat, 14 Nov 2026 10:17:56 +0100 Subject: [PATCH 3/3] Run builds outside the lockfile transaction - Split manage into three phases: record the checkout under the lockfile guard, run the build with the guard released, then record `built` only if the lock entry still points at the built commit. - Build the locked commit only; refuse a drifted checkout and point to amber_restore instead of recording a build for an unlocked HEAD. - amber_build fails without a lock entry instead of reporting "built". - exec merges opts.env over the AMBER_PLUGIN_* variables; add AMBER_PLUGIN_MODULE_DIR and AMBER_PLUGIN_DATA_DIR, and point AMBER_PLUGIN_DIR at the checkout. - Expose data_dir to build steps and document that build output must be gitignored or kept in data_dir, since restore and update refuse dirty checkouts. Note the shared budget in bulk actions. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HYR5NzbCKe2x4vjWBiynJt --- .agents/skills/rex-plugin-authoring/SKILL.md | 2 +- README.md | 11 ++- amber.lua | 79 +++++++++++++------- amber/build.lua | 18 ++++- docs/architecture.md | 6 +- tests/test.lua | 58 ++++++++++++++ types/amber.lua | 6 +- 7 files changed, 140 insertions(+), 40 deletions(-) diff --git a/.agents/skills/rex-plugin-authoring/SKILL.md b/.agents/skills/rex-plugin-authoring/SKILL.md index 69abeb2..2772170 100644 --- a/.agents/skills/rex-plugin-authoring/SKILL.md +++ b/.agents/skills/rex-plugin-authoring/SKILL.md @@ -37,7 +37,7 @@ Register actions and bindings during setup; defer operations that require a live ## Build steps and background processes -If the plugin needs work after checkout, document a `build` option for the Amber spec (a shell command, a function, or a list; see the README) rather than an install script. It runs on install and update within Amber's action budget, so keep it short. Amber holds back `setup` until it has succeeded. +If the plugin needs work after checkout, document a `build` option for the Amber spec (a shell command, a function, or a list; see the README) rather than an install script. It runs on install and update within Amber's action budget, so keep it short. Amber holds back `setup` until it has succeeded. Put build output in `.gitignore` or under `data_dir` (`AMBER_PLUGIN_DATA_DIR` for shell steps): untracked files make `amber_restore` and `amber_update` refuse the checkout. Amber has no service management and does not know where the `rex` binary is. For a long-running watcher, check the current Rex Lua reference, start it from `setup` with a detached `rex.exec.start`, resolve its script path from `ctx.dir`, keep state in `ctx.data_dir`, and make it exit when another instance is already running, because `setup` runs on every config load and check. State what you verified against a live Rex. diff --git a/README.md b/README.md index d950831..bc1970c 100644 --- a/README.md +++ b/README.md @@ -102,13 +102,16 @@ Some plugins need work after checkout, such as compiling a helper. Declare it wi }, ``` -- A string runs as `sh -c ` in the plugin directory (`module_dir` for `subdir` plugins). `AMBER_PLUGIN_NAME` and `AMBER_PLUGIN_DIR` are set in its environment. -- A function receives `{ name, dir, module_dir, commit, exec }`. `exec(argv, opts)` runs a command in `module_dir` within the remaining action time. The step fails if the function raises an error, returns `false`, or returns `nil` and a message. +- A string runs as `sh -c ` in the plugin directory (`module_dir` for `subdir` plugins). Its environment has `AMBER_PLUGIN_NAME`, `AMBER_PLUGIN_DIR` (the checkout), `AMBER_PLUGIN_MODULE_DIR`, and `AMBER_PLUGIN_DATA_DIR`. +- A function receives `{ name, dir, module_dir, data_dir, commit, exec }`. `exec(argv, opts)` runs a command in `module_dir` within the remaining action time; `opts.env` adds to the variables above instead of replacing them. The step fails if the function raises an error, returns `false`, or returns `nil` and a message. - `build = false` disables the step. Local `path` plugins cannot have one: Amber never runs anything for them. -- `amber_install`, `amber_update`, and `amber_restore` run the build when the checked-out commit differs from the one recorded as built in the lockfile (`built`). An unchanged commit is not rebuilt. `amber_build` forces it. +- `amber_install`, `amber_update`, and `amber_restore` run the build when the locked commit differs from the one recorded as built in the lockfile (`built`). An unchanged commit is not rebuilt. `amber_build` forces it. +- A build always targets the locked commit. If the checkout has moved away from it, the build is refused; run `amber_restore` first. `amber_build` needs a lock entry, so run `amber_install` first if the lockfile was removed. - A failed build still records the checkout in the lockfile, fails the action, and leaves the plugin unbuilt. Fix the cause and run `amber_build`. +- Keep build output out of Git's way. `amber_restore` and `amber_update` refuse checkouts with untracked or modified files, so list generated files in the plugin's `.gitignore` or write them to `data_dir`. - Until a build has run for the locked commit, Amber does not call the plugin's `setup`; `amber_status` reports `unbuilt`. Adding `build` to an existing plugin therefore needs one `amber_build` and a reload. If the lockfile cannot be read, Amber cannot tell and does not hold plugins back. -- Steps share the roughly 23-second action budget with Git. Anything longer fails with a timeout; start long-running work from the plugin itself instead (see [Background processes](#background-processes)). +- Steps share the roughly 23-second action budget with Git, and a bulk action shares it across all plugins. Anything longer fails with a timeout; build slow plugins one at a time with `plugin=owner/repo`, or start long-running work from the plugin itself (see [Background processes](#background-processes)). +- The build runs after the lockfile is released, so other Amber actions are not blocked by it. If the lock entry changes while it runs, the result is not recorded and `amber_build` must run again. - Build commands are trusted code with the Rex server's privileges, like plugin Lua. Read them before installing. ## Subdirectory plugins diff --git a/amber.lua b/amber.lua index f71af1f..b1cd558 100644 --- a/amber.lua +++ b/amber.lua @@ -70,19 +70,24 @@ function M.setup(config) if not plugin.local_path then by_dir[plugin.dir] = true end + plugin.data_dir = state_root and state_root .. "/" .. plugin.slug or nil by_name[plugin.name] = plugin plugins[#plugins + 1] = plugin end end + -- Runs the Git operation and records the checkout inside one lockfile + -- transaction, then runs any pending build outside it so a slow or killed + -- build never leaves the lockfile guard behind. A second transaction records + -- the build only if the lock entry still points at the commit that was built. local function manage(plugin, deadline, operation) - local build_err + local pending local result, err = lock.transaction(lockfile, function(data) local entry = data.plugins[plugin.name] if operation == "restore" and not entry then return nil, "no lockfile entry for " .. plugin.name end - local status, err + local status, op_err if operation == "build" then if plugin.local_path then return nil, "local plugins are managed outside amber" @@ -90,21 +95,26 @@ function M.setup(config) if not util.exists(plugin.dir .. "/.git/HEAD") then return nil, "not installed" end - status = plugin.build and "built" or "no build step" + if not plugin.build then + return "no build step" + end + if not entry then + return nil, "no lockfile entry; run amber_install" + end + status = "built" elseif operation == "install" then - status, err = git.install(plugin, deadline, entry and entry.commit) + status, op_err = git.install(plugin, deadline, entry and entry.commit) elseif operation == "restore" then - status, err = git.restore(plugin, deadline, entry.commit) + status, op_err = git.restore(plugin, deadline, entry.commit) else - status, err = git.update(plugin, deadline) + status, op_err = git.update(plugin, deadline) end if not status then - return nil, err + return nil, op_err end local changed = false - local hash, hash_err if operation == "update" or (operation == "install" and not entry) then - hash, hash_err = git.commit(plugin, deadline) + local hash, hash_err = git.commit(plugin, deadline) if not hash then return nil, "checkout succeeded but lockfile was not updated: " .. tostring(hash_err) end @@ -114,30 +124,41 @@ function M.setup(config) data.plugins[plugin.name] = entry changed = true end - if plugin.build and entry then - if not hash then - hash, hash_err = git.commit(plugin, deadline) - end - if not hash then - build_err = "cannot read checkout commit for build: " .. tostring(hash_err) - elseif operation == "build" or entry.built ~= hash then - local built, built_err = build.run(plugin, hash, deadline) - if built then - entry.built = hash - changed = true - else - build_err = built_err - end - end + -- The build always targets the locked commit, never whatever HEAD is. + if plugin.build and entry and (operation == "build" or entry.built ~= entry.commit) then + pending = entry.commit end return status, nil, changed end) - -- The checkout and lock entry are recorded even when the build fails, so - -- the failure is reported after the transaction and amber_build can retry. - if result and build_err then + if not result or not pending then + return result, err + end + + -- The checkout and lock entry are already recorded, so a failure below + -- leaves the plugin unbuilt and amber_build can retry. + local head, head_err = git.commit(plugin, deadline) + if not head then + return nil, "cannot read checkout commit for build: " .. tostring(head_err) + end + if head ~= pending then + return nil, "checkout differs from lockfile; run amber_restore before building" + end + local built, build_err = build.run(plugin, pending, deadline) + if not built then return nil, build_err end - return result, err + local recorded, record_err = lock.transaction(lockfile, function(data) + local entry = data.plugins[plugin.name] + if not entry or entry.commit ~= pending then + return nil, "lockfile changed during the build; run amber_build again" + end + entry.built = pending + return result, nil, true + end) + if not recorded then + return nil, "build succeeded but lockfile was not updated: " .. tostring(record_err) + end + return result end register_action("amber_install", "Install Rex Plugins", function(plugin, deadline) return manage(plugin, deadline, "install") @@ -259,7 +280,7 @@ function M.setup(config) dir = plugin.dir, module_dir = plugin.module_dir, commit = locked and locked.commit or nil, - data_dir = state_root and state_root .. "/" .. plugin.slug or nil, + data_dir = plugin.data_dir, } -- A plugin that needs a build is not set up until the build has run for -- its locked commit, since setup typically depends on the build output. diff --git a/amber/build.lua b/amber/build.lua index 0044858..2c463f4 100644 --- a/amber/build.lua +++ b/amber/build.lua @@ -24,12 +24,18 @@ end ---@return boolean? ok ---@return string? err local function run(plugin, commit, deadline) - local env = { AMBER_PLUGIN_NAME = plugin.name, AMBER_PLUGIN_DIR = plugin.module_dir } + local base_env = { + AMBER_PLUGIN_NAME = plugin.name, + AMBER_PLUGIN_DIR = plugin.dir, + AMBER_PLUGIN_MODULE_DIR = plugin.module_dir, + AMBER_PLUGIN_DATA_DIR = plugin.data_dir, + } ---@type AmberBuildContext local context = { name = plugin.name, dir = plugin.dir, module_dir = plugin.module_dir, + data_dir = plugin.data_dir, commit = commit, exec = function(argv, opts) local timeout, budget_err = remaining_budget(deadline) @@ -37,13 +43,21 @@ local function run(plugin, commit, deadline) return nil, budget_err end opts = opts or {} + -- Caller variables extend the Amber ones instead of replacing them. + local env = {} + for key, value in pairs(base_env) do + env[key] = value + end + for key, value in pairs(opts.env or {}) do + env[key] = value + end local limit = timeout if opts.timeout then limit = limit and math.min(limit, opts.timeout) or opts.timeout end return util.run(argv, { cwd = opts.cwd or plugin.module_dir, - env = opts.env or env, + env = env, timeout = limit, }) end, diff --git a/docs/architecture.md b/docs/architecture.md index b0810df..bcddd3d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -68,9 +68,11 @@ For updates, an unversioned checkout uses `git pull --ff-only`. A versioned chec ## Build steps -A managed plugin may declare `build`, normalized by `amber/spec.lua` into a list of steps. After the Git operation of `amber_install`, `amber_update`, or `amber_restore`, `manage` in `amber.lua` reads the checkout commit and runs the steps through `amber/build.lua` when the lock entry's `built` differs from it; `amber_build` skips the Git step and always runs them. String steps run as `sh -c` in `module_dir` through `rex.exec`; function steps get a context whose `exec` applies the remaining action budget. The first failing step stops the build. +A managed plugin may declare `build`, normalized by `amber/spec.lua` into a list of steps. `manage` in `amber.lua` works in three phases. First, one lockfile transaction runs the Git operation of `amber_install`, `amber_update`, or `amber_restore` (none for `amber_build`), writes the lock entry, and notes a pending build when the entry's `built` differs from its `commit`, or always for `amber_build`. `amber_build` requires an existing lock entry. Second, with the lockfile guard released, it checks that `HEAD` equals the locked commit and runs the steps through `amber/build.lua`; a drifted checkout is refused with a pointer to `amber_restore`. Third, a second transaction sets `built` to the commit, but only if the entry still locks that commit; otherwise the action fails and the build must be repeated. -The lock entry is written inside the same lockfile transaction, with `built` set to the commit only on success. A failed build still records the checkout, then the action fails, so a retry through `amber_build` has a lock entry to work with. An update to a different commit drops `built`; an update that lands on the same commit keeps it. +String steps run as `sh -c` in `module_dir` through `rex.exec` with `AMBER_PLUGIN_NAME`, `AMBER_PLUGIN_DIR`, `AMBER_PLUGIN_MODULE_DIR`, and `AMBER_PLUGIN_DATA_DIR`; function steps get a context whose `exec` applies the remaining action budget and merges caller variables over those. The first failing step stops the build. Running builds outside the transaction keeps a slow or interrupted build from leaving the `.lock` guard behind. + +A failed build leaves the recorded checkout in place with `built` unset, so `amber_build` can retry. An update to a different commit drops `built`; an update that lands on the same commit keeps it. Build output inside the checkout must be ignored by Git, since restore and detached updates refuse untracked files. At config load, an enabled plugin with a build step whose checkout exists and whose lock entry lacks `built == commit` is not set up: its load status is `unbuilt` and a warning is logged. If the lockfile cannot be read, this check is skipped. Config load never runs a build. Builds are trusted code and share the 23-second action budget with Git, so long-running work must be started by the plugin itself. diff --git a/tests/test.lua b/tests/test.lua index 2bc3684..b757ca2 100644 --- a/tests/test.lua +++ b/tests/test.lua @@ -777,6 +777,64 @@ test("rejects malformed built entries in the lockfile", function() assert(lock.read(path)) end) +test("amber_build requires a lock entry", function(amber, actions, _, calls) + amber.setup({ root = root, plugins = { { "dave/built", build = "make" } } }) + local ok, err = pcall(actions.amber_build.run, {}, { plugin = "dave/built" }) + assert(not ok and tostring(err):match("no lockfile entry"), tostring(err)) + assert(count_calls(calls, "sh") == 0, "nothing may be built without a lock entry") +end) + +test("does not build a drifted checkout", function(amber, actions, _, calls, behavior) + amber.setup({ root = root, plugins = { { "dave/built", build = "make" } } }) + actions.amber_install.run({}, { plugin = "dave/built" }) + assert(count_calls(calls, "sh") == 1) + behavior.head_hash = string.rep("b", 40) + local ok, err = pcall(actions.amber_build.run, {}, { plugin = "dave/built" }) + assert(not ok and tostring(err):match("checkout differs from lockfile"), tostring(err)) + assert(count_calls(calls, "sh") == 1, "a drifted checkout must not be built") + assert(behavior.lock_data.plugins["dave/built"].built == string.rep("a", 40)) +end) + +test("builds outside the lockfile guard", function(amber, actions, _, calls, behavior) + local guard = root .. "/test-lock.json.lock" + local function held() + local depth = 0 + for _, call in ipairs(calls) do + if call.argv[3] == guard then + depth = depth + (call.argv[1] == "mkdir" and 1 or -1) + end + end + return depth + end + local during + local move_lock = false + amber.setup({ + root = root, + plugins = { + { + "dave/built", + build = function(ctx) + during = held() + if move_lock then + behavior.lock_data.plugins["dave/built"].commit = string.rep("c", 40) + end + return ctx.exec({ "sh", "-c", "true" }, { env = { EXTRA = "1" } }) + end, + }, + }, + }) + actions.amber_install.run({}, { plugin = "dave/built" }) + assert(during == 0, "the lockfile guard must be released before building") + local build = assert(called(calls, "sh", "-c")) + assert(build.opts.env.EXTRA == "1" and build.opts.env.AMBER_PLUGIN_NAME == "dave/built") + assert(build.opts.env.AMBER_PLUGIN_DIR == root .. "/dave/built") + assert(build.opts.env.AMBER_PLUGIN_MODULE_DIR == root .. "/dave/built") + move_lock = true + local ok, err = pcall(actions.amber_build.run, {}, { plugin = "dave/built" }) + assert(not ok and tostring(err):match("lockfile changed during the build"), tostring(err)) + assert(behavior.lock_data.plugins["dave/built"].built == string.rep("a", 40)) +end) + test("rejects expired operation budgets", function() local git = assert(loadfile("amber/git.lua"))(dofile("amber/util.lua")) local status, err = git.install({ local_path = false }, os.time() - 1) diff --git a/types/amber.lua b/types/amber.lua index 650e238..04a69ad 100644 --- a/types/amber.lua +++ b/types/amber.lua @@ -21,8 +21,9 @@ ---@field name string ---@field dir string ---@field module_dir string ----@field commit string Checked-out commit being built. ----@field exec fun(argv: string[], opts?: RexExecOptions): RexExecResult?, string? Runs a command in module_dir within the remaining action budget. +---@field data_dir? string Per-plugin state directory; Amber does not create it. +---@field commit string Locked commit being built; the checkout is verified to match it. +---@field exec fun(argv: string[], opts?: RexExecOptions): RexExecResult?, string? Runs a command in module_dir within the remaining action budget; opts.env extends the AMBER_PLUGIN_* variables. ---@class AmberConfig ---@field root? string Absolute directory for managed checkouts. @@ -44,6 +45,7 @@ ---@field enabled boolean ---@field version? string ---@field build? (string|fun(ctx: AmberBuildContext): any)[] +---@field data_dir? string Per-plugin state directory, set by setup. ---@field load_status? string ---@field load_error? string