Make mission creation and cloning work in lean mode - #285
Open
CarsonDavis wants to merge 4 commits into
Open
Conversation
In lean mode there is no local Missions/ filesystem — mission assets are served from object storage — so the add handler's unconditional fs.mkdirSync under makedir always threw ENOENT. Because the Config DB row is created and committed before the mkdir runs, the mission ended up existing while the endpoint reported 'Failed to create new mission.', leaving a half-created mission that a retry reported as already existing. Gate the Missions/<name> directory creation on full mode via isLean(); full mode keeps making the directory tree exactly as before. The DB-backed mission row is still created in lean mode, just without the mkdir.
- Gate /clone on isFull() and stop its execFile callback from crashing the process: check error, guard the JSON.parse, and answer with the route's JSON failure shape; surface that message in CloneConfigModal and hide the Clone action in lean mode - Validate req.body.mission before use in add() and re-check msv is a plain object after the config deepmerge, so malformed requests get the JSON failure contract instead of an Express 500 - Create mission/Layers/Data directories with recursive mkdirSync in a try/catch: pre-existing bare mission dirs now get their subdirectories, and a failed mkdir reports honestly instead of orphaning the DB row - Lean-gate the create-directory checkbox in NewMissionModal and always send makedir:false in lean - Use the affirmative isFull() gate to match the other deployment-mode gates - Spec: pin HIDE_CONFIG, save/restore env keys, and replace the hand-rolled http helper with fetch that surfaces non-JSON bodies
- cloneLean() copies the config row (msv rename via add()) and rewrites every /assets/<old>/ URL prefix to /assets/<new>/; the python scaffold script and the hasPaths relative-path rewrite are full-mode only - Copy the mission's stored assets from assets/<old>/ to assets/<new>/ in the shared bucket via copyPrefix, which gains an optional destPrefix; a copy failure rolls back the created config row by id - Grant the admin task role s3:GetObject on the asset objects and s3:ListBucket on the assets bucket (Terraform module + policy doc), which the copy requires - Show the Clone action in lean mode again; hide the Adjust Paths checkbox there since root-relative asset URLs must not be relativized - add() passes the created row's id to its callback for the rollback - Specs cover lean clone URL rewrite + prefix copy + rollback, the full-mode script path, and copyPrefix destPrefix re-prefixing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes creating and cloning a mission work in a lean deployment, where there is no local
Missions/directory to write to.Two deployment shapes are involved throughout. Lean mode is the cloud deployment: uploaded assets live in a shared S3 bucket and the container has no
Missions/directory tree. Full mode is the traditional install, where each mission is a folder underMissions/on local disk.What this does
mkdir ./Missions/<name>on a filesystem that isn't there, so the request returned "Failed to create new mission." after the mission row had already been written — you were left with a mission the UI said had failed. Lean mode now never touchesMissions/; full mode still createsMissions/<name>,/Layers, and/Data.makediris never sent as true./assets/<oldMission>/…URL rewritten to the clone's name — absolute URLs such ashttps://…/{z}/{x}/{y}.pngare left alone — and copies the stored objects to the clone's prefix in the shared asset bucket.hasPathsto false there./addinput first. A missing or malformed mission name returns "Bad mission name." before anything else runs (a missing name used to throw), and a posted config containingmsv: nullor a non-objectmsvno longer breaks mission creation.add()'s callback result now carries the created row's id — this is what the lean clone's rollback targets.s3:GetObjectands3:ListBucketon the shared asset bucket.CopyObjectreads the source object with the caller's credentials, so without these the lean clone's copy fails with AccessDenied.copyPrefixaccepts an optionaldestPrefixthat swaps only the leading prefix of each key; omitting it keeps the previous same-key behavior.Tests cover the lean/full gating of directory creation, the lean clone's URL rewriting, prefix copy, rollback on copy failure, and no-bucket case, the full clone still invoking the Python script, and
destPrefixkey rewriting.Decisions to review
The S3 copy runs inline in the clone request.
cloneLeancreates the config row, then copies every object underassets/<old>/one at a time before responding, so clone latency grows linearly with the source mission's asset count. The alternative is a background job, or a bounded concurrent copy plus a progress-poll endpoint. This matters at a proxy or load-balancer idle timeout: the response gets cut, the copy keeps running, the client sees a failure, and the rollback deletes a config row whose assets did in fact get copied. The rollback is only clean when the failure comes from the SDK itself.Rollback policy differs between the two new paths. A failed lean clone destroys the row it created. A failed
mkdirinadd()does the opposite — it keeps the row and reports that the configuration was created but its directories weren't. Two answers to "the second phase failed" in one file. The alternative is picking one: both roll back, or both report partial success with a distinct status.Asset URLs are rewritten by string splice over the serialized config. The rewrite is
JSON.stringify(config).split('/assets/<old>/').join('/assets/<new>/'). That is safe only because the mission-name validator rejects quotes and backslashes, which makes that regex a security invariant rather than a UX nicety — loosening it later turns this into config injection. The alternative is a recursive walk that rewrites string leaves. Two side effects worth a look: the splice rewrites/assets/<old>/anywhere in the config, including prose fields and deliberate cross-mission references, and the source prefix is derived frommissionFolderName, which is user-editable free text that nothing forces to keep matching the mission name. If it was edited after uploads existed, that mission's objects are split across two prefixes and the clone copies only the current one. References to another mission's/assets/<OtherMission>/are neither rewritten nor copied, so the clone keeps reading the other mission's objects.A missing
MMGIS_SHARED_ASSET_BUCKETfails open. With the bucket env unset in lean mode, clone returns success and the config comes back with URLs already rewritten to a prefix nothing was copied into. That is harmless today only because uploads hard-fail without the bucket, so there is nothing to copy — but that is an inference about how the deployment got here, not an invariant. A bucket unset after uploads exist yields a silently broken clone. Failing closed is the alternative.Clones duplicate asset storage permanently. Every lean clone makes a full physical copy of the source's objects, and nothing reclaims them:
/destroyhas no S3 branch, and the admin task role has nos3:DeleteObjecton the asset bucket at all, so cleanup isn't possible from the admin container without an IAM change. Rollback also deliberately leaves partially-copied objects behind.assets/grows with every clone. Not taken: copy-on-write, where the clone points at the source prefix, or cleanup on delete.hasPathsis silently dropped in lean mode. The UI hides the checkbox and sends false, but the server ignores the flag instead of rejecting it, so an API caller passinghasPaths: truegets different semantics with no signal. This is deliberate — the relativize heuristic would mangle root-relative URLs — but it leaves a real gap: a config holding genuinely relative paths that don't start with/assets/is never adjusted, so the clone keeps pointing at the source mission's data.add()'s callback contract gainedid; the HTTP response did not. In-process callers can act on the exact row, which is how the lean clone's rollback avoids deleting by name. Callers of/addover HTTP can't do the same id-scoped cleanup. That split between the in-process and over-the-wire contracts for one function is intentional and worth a second opinion.Line breakdown
copyPrefixspec (40)