Add multicam image registration backend#1750
Draft
romleiaj wants to merge 27 commits into
Draft
Conversation
homography.ts: 3x3 matrix primitives (multiply, invert, apply, linear and DLT solvers) plus warp-grid subdivision helpers. transform.ts: keypointgui-style transform models (translation / rigid / similarity / affine / homography) that all estimate to a plain Matrix3, with Similarity as the single shared default. CameraCalibrationStore: the persistence core for camera-rig calibration -- per-pair homographies, correspondences, transform types, and producer provenance, with hydrate() from dataset meta, dirty tracking, and the portable calibration.json round trip (toCalibrationJson / loadCalibrationText). Interactive creation (point picking, fitting, overlay preview) is not part of this branch; it layers on top in the Manual Alignment GUI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Persist cameraHomographies / cameraCorrespondences / cameraTransformTypes / cameraCalibrationSource in dataset meta on both platforms (server allowlist + desktop meta merge), and load calibrations produced externally (e.g. by kamera) at import time: the multicam import dialog gains a per-camera transform-file picker, the desktop backend parses DIVE calibration .json pairs (fromCalibrationPairs / toCalibrationPairs) and auto-discovers a calibration.json in the import parent folder, and saveMetadata writes the portable calibration.json alongside the project so the calibration travels with the dataset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When every camera resolves a transform into the reference camera's space (first camera in display order, composed through the calibration pair graph), an Align button warps each pane's imagery and annotations into that shared space at draw time -- stored geometry always stays native per camera. Includes: AlignedViewStore + resolution wiring in the Viewer; AlignedImageLayer (sub-quad warped imagery with seam overlap); display-transform routing through every annotation layer; transform-aware linked pan/zoom (useAlignedNavigation/useLinkedViewers, with the raw screen-delta camera sync retired once transforms exist); drawing/editing on any camera with edits mapped back through the inverse; and a continuous cross-camera mirror that re-projects every geometry edit onto all other calibrated cameras. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Importing single-camera annotations into a calibrated multicam dataset now offers an "Import to all cameras" checkbox: after the import lands on the active camera, every track is copied onto each other camera through the camera-to-camera homographies (bounds, rotated bounds, and GeoJSON geometry), then saved. Unlike the Align View continuous mirror this is a data import, so confidence pairs and track/feature attributes are copied too. Gated on the whole rig resolving to the reference (alignedView.available), independent of the display toggle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dataset's saved calibration moves from a single all-pairs
calibration.json to one calibration_<camera>.json per non-reference
camera (a pair not touching the reference files under its right
camera), matching the per-camera files users already handle from
kamera. File names are discovery/provenance only: the pair's left/right
names inside each file body stay authoritative, so a misnamed or copied
file can never rebind a transform to the wrong camera. The legacy
single file is still read and is migrated (rewritten per-camera, then
removed) on the next save; stale per-camera files are cleaned up so the
on-disk set always mirrors the saved calibration.
Per-file producer stamps are compared when the set is merged (project
load and import seeding): agreement keeps the stamp, disagreement
becomes a { mixed: true, files: {...} } composite that the Align View
tooltip surfaces as a mixed-generation warning instead of composing
silently. A save never writes the composite into per-camera files --
that would read back as a unanimous rig.
Parent-folder transform discovery now returns every self-identified
calibration file (calibration.json first, then per-camera files), and
the multicam import dialog auto-attaches calibration_<camera>.json to
its matching camera slot. Files written by DIVE now also self-identify
with type: dive-camera-calibration so discovery recognizes them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-camera calibration_<camera>.json files are now the only on-disk form: the single all-pairs calibration.json is no longer read, migrated, or given priority during import discovery. The Export menu gains explicit calibration export -- one file per calibrated camera, or all of them zipped -- built from the same per-camera grouping the save path uses, and the multicam Everything zip now carries the files too. Exports resolve the calibration the same way loading does (standalone files first, the import-time meta seed otherwise) and never stamp a file with a mixed composite source. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Calibration export is now strictly per-camera -- one calibration_<camera>.json at a time, no all-cameras zip -- and works on both platforms: desktop writes through the backend as before, web builds the file client-side from the dataset meta and downloads it. The Import menu gains a matching per-dataset calibration import (same workflow as annotation import): pairs the file names replace the current ones, other cameras' pairs are kept, so per-camera files can be imported one at a time to assemble a rig; the viewer store rehydrates immediately so the Align View picks up the new transforms without a reload. Grouping and merge semantics live in a new shared vue-media-annotator module used by the desktop backend, both Export menus, and both import paths, so listed cameras, written files, and merge behavior cannot drift between platforms. Web needs no server changes: the calibration meta fields were already allowlisted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Import and Export buttons now advertise "Supplementary Data" in their hover tooltips instead of "Annotation Data", since both menus carry calibrations as well. The Import Calibration blurb shrinks to one line. The warp-on-import checkbox moves out of the active-camera alert to sit beside Overwrite as "Warp to All", always visible on multicam datasets and greyed out until the whole rig resolves, with an "N/M cameras calibrated" hint. The viewer publishes that count into AlignedViewStore from the same computation as the Align View tooltip, so the two indicators can never disagree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generic Import Calibration button becomes one button per
non-reference camera pair ("Import eo -> ir"), each scoped to its
camera: only the file's pairs naming that camera are taken, they
replace that camera's current pairs, and other cameras' pairs are
kept -- so per-camera files assemble a rig one at a time and a
multi-pair file pointed at one camera doesn't drag the rest along.
Scoping to a camera the file doesn't name refuses with a clear error.
Buttons are green when the camera already has a calibration and yellow
when it doesn't, and importing onto an existing one asks for
confirmation before replacing it, so no separate overwrite toggle is
needed. The camera filtering and pair/camera summary live in the
shared calibration-file module so desktop and web behave identically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ormat
"Calibration" collided with the pre-existing stereo camera file (an
intrinsic/extrinsic camera model for the measurement pipeline), and in
computer vision this process is image registration, so the feature says
registration everywhere: "Import Registration" / "Registration: <camera>"
menu sections, importCameraRegistration and exportCameraRegistration API
members and IPC channels, registrationProgress ("N/M cameras registered")
published on the aligned-view store, matching error messages, the
internals (CameraRegistrationStore, cameraRegistrationFiles, and the
load/merge/build helpers), and the wire format: the dataset meta field is
cameraRegistrationSource, renamed through the server models, allowlist,
and tests alongside the RegistrationSource type and stamp helpers. Files
are named <camera>_to_<reference>_registration.json and self-identify as
dive-camera-registration: each file states the destination of the mapping
it carries (ir_to_eo_registration.json registers ir onto eo), making the
reference-camera rule -- first camera in display order -- legible from
the artifacts themselves. The name scheme contains neither "calibration"
nor "cal", so these files can no longer be mis-suggested as stereo camera
files by the stereo parent-folder discovery heuristic. The Align View
keeps its name (registration is the data and process; the Align View is
the display mode that renders registered imagery), and the stereo
camera-file surfaces are untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The registration reference was implicitly the first camera in display
order, which merely coincided with the camera users pick at import in the
common EO-first case. The import dialog's picker is now labeled
"Reference Camera" ("Choose which camera to use as the reference camera
for image registration") and its choice -- stored as the dataset's
defaultDisplay -- actually drives the reference, through a shared
referenceCameraName() resolver used by the viewer's transform resolution
and N/M-registered status, both Export menus, and the desktop per-camera
file persistence, falling back to display order when the choice is
missing. The viewer now publishes the reference on the aligned-view store
even before the rig resolves, so the per-pair import buttons name it
("Import ir → eo") instead of assuming camera-map order.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
romleiaj
force-pushed
the
sealtk/image-registration
branch
from
30ebbee to
a6d545d
Compare
BryonLewis
requested changes
Jul 10, 2026
BryonLewis
left a comment
BryonLewis
left a comment
Collaborator
There was a problem hiding this comment.
These comments are for myself to go back and look more closely at. If you want to implement in Claude let me know, if not I'll work on it.
Most of the comments are structural to reduce large files and better organize this code into some folders to separate the feature as well as add some README.md files to them to give a basic outline of what the hundreds of new lines do.
A file named *_registration.json with a pairs list now qualifies without the self-identifying type marker -- the conventional name is the producer's declaration of intent, matching every other load path where type is optional. Non-conventionally-named JSONs still require the exact type string so stray JSON in the collect root is never grabbed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pair bodies are authoritative on load, so a pair naming a camera the dataset doesn't have imports fine but never resolves in the Aligned View. beginMultiCamImport now collects a warning per transform file whose pairs name cameras outside the dataset, carried on the import response as importWarnings and shown in the import confirmation dialog so the mismatch is visible before Finish Import. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Web parity for the desktop-only transform seeding: the import dialog's per-camera transform pickers and parent-folder auto-discovery now work on web (Files stashed in the multicam registry, discovery scanning the selected folder's root with the same qualification rules as the desktop backend), and the upload flow parses the attached registration files up front, seeds the new dataset's saved camera registration through the meta PATCH, and prompts the same unknown-camera warning as desktop. mergeRegistrationSources moves to the shared cameraRegistrationFiles so both platforms compose producer stamps identically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each collect's root is scanned for DIVE registration .json files (the same qualification rules as the single import's parent-folder discovery) on both platforms, and the shared batch scan assigns them to camera slots on the collect's importArgs via the slot-matching rule now extracted from the import dialog. The batch table shows what was attached, warns when a file has no free slot, and surfaces the unknown-camera seeding warnings per collect row -- batch imports skip the confirmation dialog that normally shows them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The warp renders a snapshot of the camera's displayed image element and only re-rendered on transform or frame changes, so a same-frame presentation change -- the percentile-stretch URL remap swapping the quad's <img>, or the first enhancement toggling the CSS filter -- left the warped pane stale until the Aligned View was exited. The media controller now exposes an imageRevision counter that the annotators bump on those redraws, and LayerManager re-renders the warp when it moves. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The web and desktop discovery paths each carried their own copy of the registration-file qualification rule, candidate ordering, and file-name pattern; share them from dive-common/registrationParentFolder. The unknown-camera import warning was likewise duplicated between the desktop and web seeding paths; share it as unknownCameraWarning next to mergeRegistrationSources, with desktop deriving named cameras from the pairs' own left/right fields instead of re-splitting transformTypes keys. Drop the mergeRegistrationSources re-export shim in favor of importing it from its home in cameraRegistrationFiles. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AlignedImageLayer polled with a bounded requestAnimationFrame loop to catch the annotator swapping its <img> element asynchronously, because no event reached the layer. imageRevision is now that event: drawImage bumps it on every media-quad redraw (including the post-seek swap), and the LayerManager watch re-renders the warp from it. VideoAnnotator now bumps it on the initial video quad draw, the one element swap a video pane ever makes. This also removes the poll's failure mode: it gave up after ~60 frames, so an image that loaded slower than a second left a stale warp until the next trigger; the revision bump fires whenever the image actually lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The LayerManager watch that re-renders the Aligned View warp on imageRevision bumps called alignedImageLayer.update(), but alignedImageLayer was never destructured from useLayerManagerAlignedView -- it sat inside the ...alignedViewHelpers rest object, so the callback threw on every bump and Vue swallowed the error. The warp therefore never followed image-enhancement changes (percentile-stretch toggles left it stale until the Aligned View was re-toggled); the removed rAF poll had been masking the dead watch for post-seek swaps. Move the watch into setupDisplayTransformWatches, where the layer is genuinely in scope and the transform/frame/revision triggers live together, and pin the contract with a composable-level spec (bump imageRevision with a swapped element -> the warp re-renders from it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.