Add mocap capture foundation and pure analysis pipeline#18
Draft
rupertgermann wants to merge 29 commits into
Draft
Add mocap capture foundation and pure analysis pipeline#18rupertgermann wants to merge 29 commits into
rupertgermann wants to merge 29 commits into
Conversation
Establish the canonical mocap domain vocabulary and the architectural decisions that issue #9 and follow-ups build on. - CONTEXT.md: glossary entries for CapturePerspective, StrokeSegmentationSource, MocapSession, CueLatencyBand, PoseFrameStream, PostureFault (v1 catalog), FaultThresholds, Calibration. - ADR-0001: store raw PoseFrameStream as a binary blob, not Postgres JSONB. Drops the per-frame Prisma table from the PRD draft. - ADR-0002: defer sidecar contract to Phase 2. v1 = browser-2D only. - ADR-0003: run analysis pipeline in the browser. WebSocket /api/mocap/live is persistence-only; server runs no analysis during capture. - ADR-0004: cloud-AI payload tiering for mocap data. - PRD: append "Resolved Decisions" section linking ADRs and locking v1 ship scope (US 1-11, 13, 19-30, 35, 36 ship; US 12, 14-18, 31-34 deferred to Phase 2). These are referenced by issue #9 and must land before any mocap implementation diff can be reviewed coherently. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PR1 of issue #9: lay the foundation for browser-path webcam capture. Capture worker, append endpoint, and capture UI follow in PR2/PR3. - Prisma: add MocapSession model with unique rowingSessionId for the bidirectional-exclusive link mandated in CONTEXT.md. - src/lib/mocap/storage.ts: MocapStorage interface (path generation, byte-range read, append, delete, exists) per ADR-0001 storage contract. LocalDiskStorage impl writes under storage/mocap/<userId>/ <sessionId>/ and guards against path traversal. Vercel Blob impl is stubbed behind the same interface and gated on MOCAP_STORAGE_BACKEND. - src/lib/mocap/poseFrameStream.ts: 32-byte header + 404-byte frames (33 keypoints * x,y,confidence Float32 + timestamp Float32 + qualityFlags Uint32). Reader rejects unknown magic / format / schema versions. frameCount uses an OPEN_FRAME_COUNT sentinel during streaming append, derivable from blob size on tab-close truncation (acceptance: no partial-frame corruption). - POST /api/mocap/sessions: validates source/perspective with zod, creates the row, allocates storage paths, writes the header into the pose-stream blob, returns id + paths. - DELETE /api/mocap/sessions/[id]: cascades to both blobs via the storage abstraction. - GET /api/mocap/sessions/[id]: read access used by PR2/PR3. Refs #9. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…king, AI payload tiers - #13: session list + replay pages, video streaming with range support, skeleton overlay, timeline/fault markers, freeze-at-catch/finish - #15: link/unlink MocapSession↔RowingSession API, csv-aligned re-analysis, auto-prompt on CSV import overlap, has-mocap badge in session list - #16: reanalyze API, record-only capture toggle (skip analysis), delete/reanalyze buttons in session list, run-analysis CTA in replay empty state - #17: PostureAIPayload tiers (tier 3 fault summary default, tier 2 metrics opt-in, tier 1 raw hard-walled), mocapDetailedAIShare setting, aiAnalysis.ts integration, hard-guard unit test Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…settings UI - aiAnalysis.ts + cloudAI.ts: attach PostureAIPayload to insight prompts (tier 3 default, tier 2 opt-in, tier 1 hard wall) - sync page: show overlap banner linking mocap↔rowing sessions after CSV import - overlap-check API: find unlinked MocapSessions within ±2 min of newly-imported RowingSession - settings: mocapDetailedAIShare toggle with tier explainer, disabled when cloudAIEnabled=false - tests/aiPayload.test.ts: 11 unit tests covering all tier permutations + hard-guard assertion Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- CoachingAdvisor pure module: getCoachingCues() maps faults to cue copy, audio hints, drills per fault key; deduplicates by type keeping highest severity; suppresses info by default - Capture page: live coaching cue banner after finalize, audio mute toggle, SessionCoachingSummary component showing top-3 fault types by frequency×severity with drill suggestions - MocapPreferences (verbosity/audioEnabled) added to UserSettings schema, settings service, and DB mapping Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Schema declared mocapPreferences Json? but no migration created it, causing /api/settings to 500 with Prisma P2022 ColumnNotFound. Added migration 20260508170000_add_mocap_preferences_settings.
- LiveCoachingEngine drives the pure StrokePhaseSegmenter → PostureMetricsCalculator → PostureFaultDetector pipeline incrementally; emits one CoachingCue per newly-completed stroke with per-faultType throttling. - cueAudio wraps speechSynthesis with cancel-and-replace + SSR guards. - /mocap capture page wires the engine, persists verbosity + audio preferences via mocapPreferences, auto-dismisses visual cues. - Settings: surface Cue Verbosity + Audio Cues controls. - Fix Zod settings schema to accept mocapPreferences and mocapDetailedAIShare (pre-existing silent-reject bug). - Tests: 5 fixture-driven LiveCoachingEngine cases.
…ap, sync, and scripts
- Add degraded mode functionality for handling incomplete mocap data - Implement stroke replay comparison feature with fault analysis - Add UI components for comparing fault-heavy vs clean strokes - Enhanced API routes with degraded mode handling - Add comprehensive test coverage for new functionality - Include skeleton canvas visualization for catch/finish phases
Add ADR-0005 with explicit decisions on PoseFrameStream v2 schema (keypointSchemaVersion 1->2, optional z field, world-mm-3d coordinate space), WebSocket sidecar wire protocol, three sidecar-3d fault unlocks, and privacy tier implications. Add representative freemocap output sample doc and AFK-agent implementation notes. Update CONTEXT.md glossary with v1/v2 PoseFrameStream shapes, sidecar-3d fault catalog, and split Calibration entry for browser vs Charuco. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tion (closes #24) Introduce strokeAlignment.ts with a candidate-delta cross-correlation algorithm that matches pose-derived strokes to StrokeData rows by elapsed time rather than array position. Handles different stroke counts via greedy 1:1 matching within a 2500ms tolerance window. Unmatched strokes get strokeDataId=null and csvMatchOffsetMs=null in phaseBoundariesJson. Replay header now displays "CSV-aligned" or "pose-segmented" based on the first stroke's segmentationSource. 10 fixture-driven tests cover aligned, offset, count-mismatch, noise, and one-to-one constraint cases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…oses #25) Wire stored mocap posture data into cloud AI insight generation via the existing 3-tier privacy policy. fetchPosturePayload() in useAIInsights calls the new /api/mocap/posture-summary route, builds the tiered PostureAIPayload, runs the hard keypoint guard, and passes the payload to cloudAI.generateInsights(). CloudInsight gains a posture category tag detected by keyword scan in parseInsightResponse(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Surface per-fault-type frequency trends from linked mocap sessions on the dashboard. Low-quality sessions are marked rather than excluded, with tooltip and legend callouts. Empty states explain the mocap session requirement. - Pure aggregation fn in postureTrendAggregation.ts (18 new tests) - GET /api/mocap/posture-trend route - PostureFaultTrendCard component with recharts line chart - Placed on dashboard after PeriodComparisonStats Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
#27) Add server-side /api/chat/posture-context route that aggregates fault summaries and per-stroke metrics from linked ready MocapSessions and applies the 3-tier cloud data policy (Tier 1 hard-wall when cloudAI disabled, Tier 3 fault summary by default, Tier 2 scalar metrics when mocapDetailedAIShare is enabled). The hard keypoint guard runs server-side before any payload leaves the DB. useChat.sendMessage fetches posture context before each AI call and passes the cloud-safe payload to cloudAI.sendChatMessage, which injects it into the chat system prompt. Raw keypoints, landmarks, pose-stream blobs, and video bytes are never included. Missing or unlinked mocap data is handled gracefully — the fetch is non-blocking and a null payload simply results in no posture context being added. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add PlanPostureGoal schema model (one optional goal per plan) - New API: GET/PUT/DELETE /api/training-plans/[id]/posture-goal - computePostureGoalProgress() derives current fault rate from linked mocap sessions - Plans page: optional posture goal picker in creation form + progress card in active plan view with empty state when no mocap sessions linked - 15 new tests covering progress calculation edge cases Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Award posture-clean-catch fires when a mocap-linked session has <=10% rounded-back-at-catch faults with >=20 strokes and passing quality score. Evaluation reads PostureFault derived rows via SessionFaultInput; raw pose frames are never touched. Poor-quality captures (score < 0.5 or quality flags) are excluded from eligibility. PostureFaultTrendCard triggers the check after fetching trend data and the store merges/persists the award through the existing notification flow. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e, UI toggle (closes #30) Implements the minimal tracer bullet for freemocap sidecar capture per ADR-0005. - poseFrameStream: keypointSchemaVersion v2 (x,y,z,confidence per keypoint), encodeFrameV2, header gains coordinateSpace/cameraCount; v1 API unchanged - poseFrameStreamAdapter: branches on v2 header, z-channel via keypointQuadsToPosePoints - analysis/types: z? on PosePoint, coordinateSpace/cameraCount on PoseFrameStream, Sidecar3DMetrics, three new fault types, 'pending' severity - postureMetrics: toProjectedStream coordinate-space adapter (world-mm-3d to normalized-2d projection for v1 fault rules), computeSidecar3DMetrics - postureFaultDetector: three fault rule stubs emitting severity=pending - sidecarClient: WebSocket + health poller for localhost:8765 - API routes: POST/GET /sidecar/connect|status per session - sessions route: accepts source=sidecar, capturePerspective=sidecar-3d - Prisma: calibrationId, cameraCount on MocapSession; sidecarPort on UserSettings - Capture page: Multi-camera sidecar toggle with health poll - coaching: drill/copy/severity entries for three new fault types - Tests: 13 new tests (v2 codec, fixture, adapter, fault stubs) - Docs: sidecar-local-setup.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Testing
npm run testpassed./node_modules/.bin/tsc --noEmit --pretty falsepassed