Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Final Report: LearningAI architecture books admin brain and revision cleanup

## Outcome

The architecture library, learner Admin view, fullscreen tutor flow, and
learning-book generation contract now describe and expose the same product
model. The copy is clearer, implementation status is explicit, and the UI uses
existing data rather than invented analytics.

## Accepted Results

- Rewrote the Tutor System Architecture and User Brain Architecture books with
clearer chapter structure, linked sources, expanded terminology, and honest
completion boundaries.
- Reframed local beta controls as operator controls and explained why each
retained control exists.
- Made the Admin learner view the default, reduced the primary navigation to
four decision-useful sections, and added per-learner concept and evidence
inspection.
- Added fullscreen tutor chat with or without a PDF, including responsive
mobile behavior.
- Changed generated learning-book guidance and fallback material to produce
explanations, relationships, worked examples, diagrams, questions, and code
when the subject requires it.
- Prevented stale chapter audio from attaching to a rewritten chapter title.

## Rejected Results

- Did not claim that aspirational brain contracts are fully enforced by the
runtime.
- Did not fabricate secure tenant identity or server-wide learner analytics;
the current Admin selector truthfully uses locally stored learner names.
- Did not delete ambiguous untracked directories or unrelated dirty-worktree
edits.

## Conflicts Resolved

- Preserved the existing deep Admin debugging routes under a collapsed
advanced section so operational tests and specialist workflows remain
available without dominating the main dashboard.
- Kept the established Dexie and learning-book schemas unchanged to avoid an
unnecessary data migration.

## Verification Evidence

- Full test program: 275 Node plus 591 DOM tests, 866 total, passed.
- Type checks, production build, formatting, whitespace checks, and the
Graphify scratch guard passed through `brain:postchange`.
- Desktop and 390x844 mobile Chrome checks passed for the revised library,
status chapter, Admin learner view, design component map, operator controls,
and no-PDF fullscreen chat.
- The local application returned HTTP 200 at `http://127.0.0.1:3001/`.

## Remaining Risks

- Per-learner Admin data is local-device data keyed by a display name, not
authenticated multi-tenant storage. Production use still requires durable
user IDs, authorization, row-level isolation, consent, and audit controls.
- Rewritten chapters intentionally ignore old title-mismatched audio. New audio
assets should be generated before considering every chapter audio-ready.
- Some runtime brain behavior remains less strict than the documented target;
the books label those areas as partial or deferred.

## Reusable Follow-up

- Use this report and `state.json` as the acceptance record for future
multi-user persistence, audio regeneration, and evidence-gating work.
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Orchestration: LearningAI architecture books admin brain and revision cleanup

## Execution Rules

- Keep the original objective intact.
- Ask for approval before risky, expensive, external, or destructive actions.
- Keep immediate blocking work local.
- Delegate only bounded, disjoint, materially useful packets.
- Integrate packet results before final verification.

## Branching Rules

- If book copy overstates runtime completion, rewrite it as accepted direction
plus explicit implementation status.
- If a requested Admin metric has no real data source, do not fabricate it;
show an unavailable/needs-instrumentation state.
- If a cleanup candidate is referenced, persisted, or user-owned, retain it.
- If browser access remains blocked, use the existing rendered tests and a
local Playwright fallback and record the limitation.

## Packet Prompts

### Packet 1: Books

Rewrite built-in architecture content for clear technical reading, linked
citations, glossary coverage, and truthful completion status.

### Packet 2: Admin

Reduce Admin to essential operational and learner-analysis views. Add readable
guidance and per-user brain inspection using real stored data.

### Packet 3: Chat and Learning Books

Enable no-PDF fullscreen chat and improve generated learning-book material with
explanations, examples, diagrams, and code when appropriate.

### Packet 4: Cleanup and QA

Audit changed and directly connected files, remove only proven dead code, and
run focused plus global verification.

## Completion Audit

- Every success criterion has source or test evidence.
- No unrelated dirty-worktree changes were reverted.
- Remaining runtime gaps are reported plainly.
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# LearningAI architecture books admin brain and revision cleanup

## Goal

Turn the Revision library, architecture books, Admin dashboard, fullscreen chat,
and generated learning books into one coherent product that a motivated
teenager can understand without reducing the technical depth.

## Success Criteria

- Tutor System Architecture chapters are reorganized and rewritten in clear
language, with an explicit current-state/completion chapter.
- User Brain Architecture chapters are renamed and rewritten around the actual
learner model, evidence boundaries, goals, and implementation status.
- App Design Language component maps have readable spacing, linked citations,
a complete glossary, and a clear explanation or removal of local beta
controls.
- Admin shows only decision-useful views and explains what each view means.
- Admin can select a learner and inspect that learner's knowledge graph,
mastery, misconceptions, and learning patterns.
- Chat can become fullscreen even when no PDF is loaded.
- Generated learning books read like revision material with explanations,
examples, diagrams, and code where the studied subject needs them.
- Cleanup removes only verified dead/obsolete code or files and preserves all
unrelated user changes.
- Focused tests, typecheck, build, postchange, and rendered UI checks pass.

## Current Context

- The worktree is already dirty from voice architecture work and other user
changes; unrelated edits must be preserved.
- Graphify MCP is attached to the wrong project. Local
`graphify-out/graph.json` is the architecture authority.
- The connected in-app browser blocks both `localhost:3001` and
`127.0.0.1:3001` with `ERR_BLOCKED_BY_CLIENT`; rendered validation must use
the repo's Playwright/test fallback unless browser access recovers.
- `brain:postchange` exists and passes. The older `brain:debug` and
`brain:ui-regression` scripts referenced by the generic debug skill are not
present in this checkout.

## Constraints

- Follow AGENTS.md: Graphify first, then directly connected files only.
- Do not regenerate or manually edit `graphify-out`.
- Keep architecture graph concepts separate from the learner-facing brain.
- Keep Admin operational and scannable rather than turning it into a marketing
page.
- Do not delete files without evidence that they are unused or generated.

## Risks

- Architecture books may describe contracts the runtime does not yet enforce.
- AdminView and RevisionView are large shared surfaces with broad tests.
- Learning-book schema and Dexie migrations are high-risk boundaries.
- Fullscreen chat must not break PDF study mode or mobile layout.
- Broad cleanup could accidentally remove user-owned worktree changes.

## Approval Required

- No external writes or deployment.
- Destructive file deletion is gated by an evidence-backed cleanup report. No
ambiguous files will be deleted automatically.

## Work Packets

1. Architecture books and citations:
`src/lib/tutorBook.json`, `src/lib/userBrainArchitectureBook.ts`,
`src/views/RevisionView.tsx`, architecture readiness tests.
2. Design language and component map:
`src/views/RevisionView.tsx` and directly connected design components.
3. Admin simplification and per-user brain:
`src/views/AdminView.tsx`, learner-memory read APIs, navigation tests.
4. Fullscreen chat:
`src/views/StudyView.tsx`, `src/components/ChatPanel.tsx`, store/types and
rendered study-flow tests.
5. Generated revision books:
`src/memory/memory.orchestrator.ts`, learning-book types, Revision rendering,
and focused memory tests.
6. Cleanup and verification:
changed-work audit, dead-code evidence, tests, build, postchange, and
Playwright fallback.

## Integration Policy

Prefer existing schemas, state, components, and visual language. Keep edits
inside established ownership boundaries. Integrate only changes that are
supported by live source and focused tests.

## Verification

- Focused source-contract and rendered tests per packet.
- `npm run test:node`
- relevant Vitest DOM tests
- `npm run lint`
- `npm run build`
- `npm run brain:postchange -- --reason architecture-admin-revision-cleanup`
- rendered desktop/mobile Playwright checks with console and overflow evidence.

## Reusable Artifacts

- This workflow directory.
- A concise final report with accepted changes, cleanup evidence, and remaining
implementation gaps.
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Results

## Books

- Tutor architecture: 13 rewritten chapters with current-state and release
chapters.
- User brain architecture: 8 rewritten chapters with a substantial glossary
and linked references.
- App design language: clearer component map and operator-control rationale.

## Product Surfaces

- Admin defaults to learner analysis and keeps specialist diagnostics under an
advanced disclosure.
- Tutor chat supports fullscreen use without a document.
- Learning books use revision-material structure rather than transcript-like
concept lists.

## Cleanup

- Removed proven `.DS_Store` artifacts and the generated `.tmp-test`
directory.
- Preserved unrelated dirty files, untracked `docs/`, `output/`, and the
separate voice workflow because ownership or obsolescence was not proven.

## Verification

- `npm test`: 866 passed.
- `npm run lint`: passed.
- `npm run build`: passed.
- `npm run brain:postchange -- --reason architecture-admin-revision-cleanup`:
passed.
- Desktop and mobile rendered checks: passed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"title": "LearningAI architecture books admin brain and revision cleanup",
"slug": "learningai-architecture-books-admin-brain-and-revision-cleanup",
"created_at": "2026-06-08T06:15:58+00:00",
"status": "completed",
"approval": {
"required": true,
"granted": true,
"notes": "User requested broad cleanup. Ambiguous destructive deletions remain gated by evidence."
},
"packets": [
{ "id": "books", "status": "completed" },
{ "id": "admin", "status": "completed" },
{ "id": "chat-learning-books", "status": "completed" },
{ "id": "cleanup-qa", "status": "completed" }
],
"verification": {
"status": "passed",
"checks": [
"Local Graphify queries identified the connected architecture surfaces.",
"Focused architecture, audio, Admin, Study, and Revision tests passed.",
"Full npm test passed: 275 Node tests and 591 DOM tests, 866 total.",
"npm run lint passed.",
"npm run build passed.",
"npm run brain:postchange -- --reason architecture-admin-revision-cleanup passed.",
"git diff --check passed.",
"Desktop and mobile Chrome rendering checks passed for the library, architecture chapters, learner Admin, component map, operator controls, and fullscreen no-PDF chat.",
"The local server returned HTTP 200 at http://127.0.0.1:3001/."
]
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Final Report: Sub 300ms Voice Brain Architecture Research

## Outcome
Use a Tutor-owned realtime voice broker. Do not keep relying on Deepgram Voice Agent as the whole agent runtime, and do not treat MisoTTS as duplex. The new design should be:

Browser AudioWorklet -> Tutor broker WebSocket -> Deepgram STT -> GPT-4o-mini learner stream -> MisoTTS streaming/cancellable audio -> browser playback

In parallel:

Tutor broker -> GPT-5.5 background agent/tool queue -> web/search/PDF/code/tool results -> foreground insertion queue

## Accepted Results
- Thinking Machines' article supports the architecture split the user wants: a real-time interaction model stays present while a background model handles deeper reasoning/tool use and streams results back into the conversation.
- Their technical direction uses 200ms micro-turns and persistent streaming sessions. We should copy the interaction pattern, not pretend a cascaded provider stack has the same native architecture.
- Miso Labs claims 110ms realtime latency and local/on-prem deployment, but the public MisoTTS card identifies it as an 8B text-to-speech model based on Sesame CSM. It is the voice renderer, not the whole interaction model.
- Deepgram's own latency docs place streaming transcription at 150-300ms and total transcript latency at 200-500ms. This makes full user-speech-to-semantic-answer under 300ms unrealistic as a guaranteed system target with cloud STT plus cloud LLM.
- OpenRouter supports `openai/gpt-4o-mini` through an OpenAI-compatible API, making it suitable for the small foreground learner LLM.
- OpenAI's GPT-5.5 docs show streaming, function calling, structured outputs, and Responses API tools such as web search, file search, code interpreter, hosted shell, apply patch, MCP, and tool search. That fits the async background layer.
- Current LearningAI already has useful primitives: `/api/voice-agent`, browser microphone/audio playback, barge-in stopping, voice tools, `/api/tts`, and `scripts/misotts_api_server.py`.

## Rejected Results
- Rejected "Miso solves duplex." MisoTTS is TTS-only in the public model card; duplex must be implemented in the broker.
- Rejected "all output under 300ms" as a hard guarantee. The source-backed target should separate local cancellation/reaction latency from full semantic answer latency.
- Rejected a big-bang replacement. Keep the current Deepgram Voice Agent path as fallback until the new broker is proven.

## Conflicts Resolved
- The user's desired Thinking Machines-style full-duplex feel conflicts with Miso's half-duplex TTS nature. Resolution: use Miso only for audio rendering and keep STT open during playback; cancel/flush TTS on barge-in.
- The current code delegates the whole voice loop to Deepgram Voice Agent. Resolution: add a Tutor-owned `/api/voice-broker` beside it, then migrate after measured proof.

## Verification Evidence
- Graphify query routed the current voice architecture to `server.ts`, `src/components/ChatPanel.tsx`, `src/lib/voiceAgentTools.ts`, `src/lib/chatAgentTools.ts`, `src/memory/brain.context.ts`, `src/memory/brain.rehearsal.ts`, `scripts/misotts_api_server.py`, `src/memory/beta.diagnostics.ts`, and `src/views/AdminView.tsx`.
- Targeted source inspection confirmed `server.ts` proxies `/api/voice-agent` to Deepgram Voice Agent, configures Deepgram listen, GPT-4o-mini think, and Deepgram speak providers, and has `/api/tts` routing to MisoTTS.
- Targeted source inspection confirmed `ChatPanel.tsx` streams PCM16, plays binary PCM, stops active audio on barge-in, and handles voice tool callbacks.
- Targeted source inspection confirmed `scripts/misotts_api_server.py` currently generates full WAV responses under a lock.

## Remaining Risks
- Actual Miso whole-utterance output is verified on the Vast.ai machine, but it
is not yet a realtime streaming path. Cold load was 87.77s; warm whole-request
tunnel latency was 6.13s for a very short utterance.
- OpenRouter GPT-4o-mini TTFT and GPT-5.5 background latency must be measured in the chosen region.
- True Miso streaming may require modifying the Miso generator internals; a phrase-chunked bridge is the fallback.
- The current browser microphone path uses ScriptProcessor, which should be replaced by AudioWorklet for stable low-latency frames.
- Provider keys, GPU deployment, and live traffic are still pending user input.

## Reusable Follow-up
Implementation order:
1. Done locally: add `VITE_VOICE_BROKER_MODE=custom` and `/api/voice-broker` beside the current path.
2. Done locally: stage brain-context, previous-memory, active-book/document metadata, foreground model, MisoTTS, and GPT-5.5 background queue events without provider traffic.
3. Next after GPU/keys: add AudioWorklet 20ms capture for the new broker.
4. Next after keys: implement STT-only Deepgram WebSocket client with interim/final transcript events.
5. Next after keys: stream GPT-4o-mini from OpenRouter with short voice prompts and no foreground blocking tools.
6. Next after GPU: upgrade Miso service to stream/cancel audio; begin with phrase chunking because the verified wrapper is whole-utterance WAV output.
7. Next after GPT-5.5 key: replace the staged background queue with real GPT-5.5 tool execution and result insertion.
8. Add Admin diagnostics for p50/p95/p99 stage timings.

Minimal Vast.ai machine:
- 1x RTX 4090 24GB preferred, or RTX 3090 24GB if cheaper.
- 8 vCPU minimum, 12-16 preferred.
- 32GB RAM minimum, 64GB preferred.
- 100GB NVMe minimum.
- Ubuntu + CUDA 12.x + PyTorch-compatible image.

Safer but not necessary for first beta: L40S 48GB or A100 40GB.
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Orchestration: Sub 300ms Voice Brain Architecture Research

## Execution Rules

- Keep the original objective intact.
- Ask for approval before risky, expensive, external, or destructive actions.
- Keep immediate blocking work local.
- Delegate only bounded, disjoint, materially useful packets.
- Integrate packet results before final verification.

## Branching Rules
- If Miso is not full-duplex, implement duplex at the broker layer.
- If sub-300ms full answer latency conflicts with STT/LLM/TTS vendor ranges, report the realistic latency target instead of overpromising.
- If implementation requires live keys or GPU access, stop at readiness and ask for the machine/key inputs.

## Packet Prompts
- P1 external research: verify Thinking Machines, MisoTTS, Deepgram, OpenRouter, and GPT-5.5 facts.
- P2 current architecture: use Graphify first, then inspect only voice/socket/tool files.
- P3 target architecture: define broker protocol, cancellation, async background jobs, and latency budget.
- P4 implementation readiness: define GPU spec, config inputs, staged code plan, and verification gates.

## Completion Audit
- Graphify-first source discovery completed.
- Packet notes completed.
- Final report completed.
- Runtime source changes intentionally deferred until GPU/API-key inputs are available.
Loading