Skip to content

Make mission creation and cloning work in lean mode - #285

Open
CarsonDavis wants to merge 4 commits into
developmentfrom
fix/lean-mission-add-mkdir
Open

Make mission creation and cloning work in lean mode#285
CarsonDavis wants to merge 4 commits into
developmentfrom
fix/lean-mission-add-mkdir

Conversation

@CarsonDavis

@CarsonDavis CarsonDavis commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Makes creating and cloning a mission work in a lean deployment, where there is no local Missions/ directory to write to.

Two deployment shapes are involved throughout. Lean mode is the cloud deployment: uploaded assets live in a shared S3 bucket and the container has no Missions/ directory tree. Full mode is the traditional install, where each mission is a folder under Missions/ on local disk.

What this does

  • Creating a mission in lean mode succeeds. Previously the server tried to mkdir ./Missions/<name> on a filesystem that isn't there, so the request returned "Failed to create new mission." after the mission row had already been written — you were left with a mission the UI said had failed. Lean mode now never touches Missions/; full mode still creates Missions/<name>, /Layers, and /Data.
  • Hides the directory checkbox in lean mode. The "Create a /Missions/{Mission Name} directory" checkbox and its explanatory text no longer appear there, and makedir is never sent as true.
  • Reports a partial create honestly. If directory creation fails in full mode, the response now says the configuration was created but its directories weren't, instead of a generic failure.
  • Cloning in lean mode copies assets in S3 instead of running the filesystem script. The old path ran the full-mode Python script, which produced no output; parsing that empty output threw inside the exec callback and the request died with no response at all. The clone now copies the config with every root-relative /assets/<oldMission>/… URL rewritten to the clone's name — absolute URLs such as https://…/{z}/{x}/{y}.png are left alone — and copies the stored objects to the clone's prefix in the shared asset bucket.
  • Rolls back a failed clone. If the asset copy fails, the newly created config row is deleted by its id (not by name, so a concurrent save under the same name survives) and the modal shows the server's actual message, ending in "The clone was rolled back.", rather than the generic "Failed to clone this mission."
  • Cloning with no shared asset bucket configured still succeeds, with nothing copied.
  • Hides the "Adjust Paths" checkbox in lean mode and forces hasPaths to false there.
  • Full-mode cloning keeps working, and stops crashing the process. The happy path is unchanged; a failing or non-JSON clone script now returns "Failed to clone mission." instead of throwing an uncaught exception that took down Node.
  • Validates /add input first. A missing or malformed mission name returns "Bad mission name." before anything else runs (a missing name used to throw), and a posted config containing msv: null or a non-object msv no longer breaks mission creation.
  • add()'s callback result now carries the created row's id — this is what the lean clone's rollback targets.
  • Grants the admin task role s3:GetObject and s3:ListBucket on the shared asset bucket. CopyObject reads the source object with the caller's credentials, so without these the lean clone's copy fails with AccessDenied.
  • copyPrefix accepts an optional destPrefix that swaps only the leading prefix of each key; omitting it keeps the previous same-key behavior.

Tests cover the lean/full gating of directory creation, the lean clone's URL rewriting, prefix copy, rollback on copy failure, and no-bucket case, the full clone still invoking the Python script, and destPrefix key rewriting.

Decisions to review

The S3 copy runs inline in the clone request. cloneLean creates the config row, then copies every object under assets/<old>/ one at a time before responding, so clone latency grows linearly with the source mission's asset count. The alternative is a background job, or a bounded concurrent copy plus a progress-poll endpoint. This matters at a proxy or load-balancer idle timeout: the response gets cut, the copy keeps running, the client sees a failure, and the rollback deletes a config row whose assets did in fact get copied. The rollback is only clean when the failure comes from the SDK itself.

Rollback policy differs between the two new paths. A failed lean clone destroys the row it created. A failed mkdir in add() does the opposite — it keeps the row and reports that the configuration was created but its directories weren't. Two answers to "the second phase failed" in one file. The alternative is picking one: both roll back, or both report partial success with a distinct status.

Asset URLs are rewritten by string splice over the serialized config. The rewrite is JSON.stringify(config).split('/assets/<old>/').join('/assets/<new>/'). That is safe only because the mission-name validator rejects quotes and backslashes, which makes that regex a security invariant rather than a UX nicety — loosening it later turns this into config injection. The alternative is a recursive walk that rewrites string leaves. Two side effects worth a look: the splice rewrites /assets/<old>/ anywhere in the config, including prose fields and deliberate cross-mission references, and the source prefix is derived from missionFolderName, which is user-editable free text that nothing forces to keep matching the mission name. If it was edited after uploads existed, that mission's objects are split across two prefixes and the clone copies only the current one. References to another mission's /assets/<OtherMission>/ are neither rewritten nor copied, so the clone keeps reading the other mission's objects.

A missing MMGIS_SHARED_ASSET_BUCKET fails open. With the bucket env unset in lean mode, clone returns success and the config comes back with URLs already rewritten to a prefix nothing was copied into. That is harmless today only because uploads hard-fail without the bucket, so there is nothing to copy — but that is an inference about how the deployment got here, not an invariant. A bucket unset after uploads exist yields a silently broken clone. Failing closed is the alternative.

Clones duplicate asset storage permanently. Every lean clone makes a full physical copy of the source's objects, and nothing reclaims them: /destroy has no S3 branch, and the admin task role has no s3:DeleteObject on the asset bucket at all, so cleanup isn't possible from the admin container without an IAM change. Rollback also deliberately leaves partially-copied objects behind. assets/ grows with every clone. Not taken: copy-on-write, where the clone points at the source prefix, or cleanup on delete.

hasPaths is silently dropped in lean mode. The UI hides the checkbox and sends false, but the server ignores the flag instead of rejecting it, so an API caller passing hasPaths: true gets different semantics with no signal. This is deliberate — the relativize heuristic would mangle root-relative URLs — but it leaves a real gap: a config holding genuinely relative paths that don't start with /assets/ is never adjusted, so the clone keeps pointing at the source mission's data.

add()'s callback contract gained id; the HTTP response did not. In-process callers can act on the exact row, which is how the lean clone's rollback avoids deleting by name. Callers of /add over HTTP can't do the same id-scoped cleanup. That split between the in-process and over-the-wire contracts for one function is intentional and worth a second opinion.

Line breakdown

Category Lines %
Tests 579 60.3%
Production code 348 36.3%
Config/build 33 3.4%
Total 960 100%
  • Production — backend clone/add routes (250), New Mission modal (44), Clone modal (41), aws-provision copy helper (13)
  • Tests — lean clone spec (346), add-gating spec (193), copyPrefix spec (40)
  • Config/build — Terraform IAM module (19), IAM policy document (14)

In lean mode there is no local Missions/ filesystem — mission assets are
served from object storage — so the add handler's unconditional
fs.mkdirSync under makedir always threw ENOENT. Because the Config DB row
is created and committed before the mkdir runs, the mission ended up
existing while the endpoint reported 'Failed to create new mission.',
leaving a half-created mission that a retry reported as already existing.

Gate the Missions/<name> directory creation on full mode via isLean();
full mode keeps making the directory tree exactly as before. The DB-backed
mission row is still created in lean mode, just without the mkdir.
- Gate /clone on isFull() and stop its execFile callback from crashing
  the process: check error, guard the JSON.parse, and answer with the
  route's JSON failure shape; surface that message in CloneConfigModal
  and hide the Clone action in lean mode
- Validate req.body.mission before use in add() and re-check msv is a
  plain object after the config deepmerge, so malformed requests get the
  JSON failure contract instead of an Express 500
- Create mission/Layers/Data directories with recursive mkdirSync in a
  try/catch: pre-existing bare mission dirs now get their subdirectories,
  and a failed mkdir reports honestly instead of orphaning the DB row
- Lean-gate the create-directory checkbox in NewMissionModal and always
  send makedir:false in lean
- Use the affirmative isFull() gate to match the other deployment-mode
  gates
- Spec: pin HIDE_CONFIG, save/restore env keys, and replace the
  hand-rolled http helper with fetch that surfaces non-JSON bodies
- cloneLean() copies the config row (msv rename via add()) and rewrites
  every /assets/<old>/ URL prefix to /assets/<new>/; the python scaffold
  script and the hasPaths relative-path rewrite are full-mode only
- Copy the mission's stored assets from assets/<old>/ to assets/<new>/
  in the shared bucket via copyPrefix, which gains an optional
  destPrefix; a copy failure rolls back the created config row by id
- Grant the admin task role s3:GetObject on the asset objects and
  s3:ListBucket on the assets bucket (Terraform module + policy doc),
  which the copy requires
- Show the Clone action in lean mode again; hide the Adjust Paths
  checkbox there since root-relative asset URLs must not be relativized
- add() passes the created row's id to its callback for the rollback
- Specs cover lean clone URL rewrite + prefix copy + rollback, the
  full-mode script path, and copyPrefix destPrefix re-prefixing
@CarsonDavis CarsonDavis changed the title Skip mission-directory creation in lean mode Make mission creation and cloning work in lean mode Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant