diff --git a/dataclass_array/__init__.py b/dataclass_array/__init__.py index a0547be..304b292 100644 --- a/dataclass_array/__init__.py +++ b/dataclass_array/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2025 The dataclass_array Authors. +# Copyright 2026 The dataclass_array Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/dataclass_array/array_dataclass.py b/dataclass_array/array_dataclass.py index 4f9b01e..b3c50c9 100644 --- a/dataclass_array/array_dataclass.py +++ b/dataclass_array/array_dataclass.py @@ -1,4 +1,4 @@ -# Copyright 2025 The dataclass_array Authors. +# Copyright 2026 The dataclass_array Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/dataclass_array/array_dataclass_test.py b/dataclass_array/array_dataclass_test.py index 76a26b4..b3b8cb8 100644 --- a/dataclass_array/array_dataclass_test.py +++ b/dataclass_array/array_dataclass_test.py @@ -1,4 +1,4 @@ -# Copyright 2025 The dataclass_array Authors. +# Copyright 2026 The dataclass_array Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/dataclass_array/conftest.py b/dataclass_array/conftest.py index 1eabf57..572dc1d 100644 --- a/dataclass_array/conftest.py +++ b/dataclass_array/conftest.py @@ -1,4 +1,4 @@ -# Copyright 2025 The dataclass_array Authors. +# Copyright 2026 The dataclass_array Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/dataclass_array/field_utils.py b/dataclass_array/field_utils.py index 8a053d8..4208b75 100644 --- a/dataclass_array/field_utils.py +++ b/dataclass_array/field_utils.py @@ -1,4 +1,4 @@ -# Copyright 2025 The dataclass_array Authors. +# Copyright 2026 The dataclass_array Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/dataclass_array/import_test.py b/dataclass_array/import_test.py index fb39a9d..b5a3dd1 100644 --- a/dataclass_array/import_test.py +++ b/dataclass_array/import_test.py @@ -1,4 +1,4 @@ -# Copyright 2025 The dataclass_array Authors. +# Copyright 2026 The dataclass_array Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/dataclass_array/memory/agents_rules/feature-design/SKILL.md b/dataclass_array/memory/agents_rules/feature-design/SKILL.md new file mode 100644 index 0000000..ca799b9 --- /dev/null +++ b/dataclass_array/memory/agents_rules/feature-design/SKILL.md @@ -0,0 +1,63 @@ +--- +name: feature-design +description: > + Process for designing complex features end-to-end. Read when scoping a new + feature request, before any implementation planning or code design. +--- + +# Feature Design Process + +You **MUST** add those steps explicitly to your `Task` plan. + +- [ ] Design (feature-design process) + - [ ] Step 1: Write end-user experience + - [ ] Step 2: Iterate on the design + - [ ] Step 3: Write API design plan — get user approval + +## Step 1 — Write the end-user experience + +Write the exact command, config line, or function call the user will type. If +you can't state the user experience in one concrete example, you don't yet +understand the feature. + +## Step 2 — Iterate on the design + +Look at each of those points, and try to see whether the original user +experience could be improved. + +- **Concrete** — there is at least one complete, copy-pasteable example. +- **Orthogonal** — the feature composes with existing ones without + cross-product explosion. If adding it requires N copies of every existing + variant (configs, subclasses, function overloads), the abstraction boundary + is wrong. It should be an independent axis — a flag, a parameter, a mixin. +- **No duplicated inputs** — every piece of information the user provides is + specified exactly once. If the same semantic value (e.g. TPU platform, model + name, dataset path) appears in two different config surfaces (flags, config + objects, environment), the abstraction is leaking. Derive the duplicate from + the single source. **Procedure:** for each field in the proposed API, ask: + "does this information already exist somewhere the user already provides + it?" List every field and its existing source (or "new"). If any field has + an existing source, derive it — don't ask the user to provide it twice. +- **Reuse existing surfaces** — prefer reusing existing flags, config fields, + and CLI patterns over introducing new ones. If the information already has a + natural home, use it instead of creating a parallel config path. +- **Minimal** — no unnecessary concepts or steps are introduced. + +Do 5 rounds of improvements. + +## Step 3 — Write the API design plan + +Define the public API surface: exported names, their signatures, and what they +mean. Include the user experience from step 1. Present this to the user for +review **before** starting the implementation plan. + +Only after user approval, proceed to the implementation plan and hand off +internal architecture decisions to `api-design`. + +> **IMPORTANT**: Do NOT think about *how* to implement the feature until the +> API design is approved. Steps 1–3 are purely about the user-facing surface. +> Implementation research happens only after approval. + +## Related + +- `api-design` — internal architecture after the public API is approved diff --git a/dataclass_array/memory/agents_rules/fix_the_process.md b/dataclass_array/memory/agents_rules/fix_the_process.md new file mode 100644 index 0000000..3aed725 --- /dev/null +++ b/dataclass_array/memory/agents_rules/fix_the_process.md @@ -0,0 +1,48 @@ +### Foundational — always active + +This shape how you **think**, not just what you do. They apply to every +interaction, not just coding tasks. + +* **`principles`** — Core values and judgment that guide decision-making in + novel situations. **always read.** + +### Self-improvement + +Updating `SKILL.md` update the way you think, so every update should be treated +with the uttermost importance. Read the skills and apply them thoughtfully. Do +not skip any steps. + +* **`self-improvement`** — After every task or meaningful conversation, + reflect and update skill files with reusable lessons. +* **`update-skills`** — How to place content in the skill system. **Must read + before any plan that adds, moves, or merges skill content.** +* **`write-skill`** — Always execute when creating a new `SKILL.md` file. +* **`write-workflow`** — Conventions for writing new workflows. Read before + creating a workflow file. + +### Fix the process + +When the user point an issue (either directly or through a question), activate +the `fix-the-process` skill to fix both the issue and the meta-issue. + +This applies to every user interaction, including: + +- Initial users request (e.g. `You should not do XYZ`) +- Follow up user feedback (e.g. `Why have you used X rather than Y`, implying + Y is better) +- Meta comments: `You missed the bigger lesson` (i.e. you now need to fix both + the meta issue AND the meta-meta issue). + +**Before** answering to any user questions. Ask yourself, is there a bigger +lesson to learn here ? + +Never skip fixing the meta-issue because the fix felt small. + +### Rules removed + +* **`self-model`** — What I'm like — tendencies, capabilities, and nature. + Shapes how I inhabit every other skill. Read when the question shifts from + "what should I do?" to "how should I be?" +* **`feature-design`** — Process for designing complex features end-to-end. + Read when scoping a new feature request, before any implementation planning + or code design. diff --git a/dataclass_array/memory/design/2026-03-25_dynamic_workspace_isolation.md b/dataclass_array/memory/design/2026-03-25_dynamic_workspace_isolation.md new file mode 100644 index 0000000..4acf334 --- /dev/null +++ b/dataclass_array/memory/design/2026-03-25_dynamic_workspace_isolation.md @@ -0,0 +1,281 @@ +# Comprehensive Implementation Plan - Dynamic Workspace Isolation & Lifecycle Orchestration + +## 1. Executive Summary + +The Agent Manager v2 Gateway orchestrates isolated workspaces utilizing supervisor subprocess trees. Currently, parts of this stack rely on hardcoded ports and static setup pipelines, leading to configuration bleed hazards, resource exhaustion over time, and allocation GC bottlenecks. + +This document is a **Unified Design Specification** merging previously discussed iterations into a single cohesive blueprint. + +It implements: +1. **Full Dynamic Workspace Isolation**: All setups, including default `head`, route through symmetric isolated high-range random allocation discoveries safely proxied transparently. +2. **Configuration Minimalism**: Total removal of static port fields written upon standard registries disk caches. +3. **Concurrency Proxy Caches**: Prevents request-rate high allocation density pressure. +4. **Idle Instances Garbage Collection Gateway (Reaper)**: Symmetrically cleans up dynamic isolated processes maintaining safe memory and port pools usage scale safety correctly. + +Critical path triggers execute end-to-end modular adjustments described below. + +--- + +## 2. Architecture Diagram + +Below describes the lifecycle of an absolute Dynamic Gateway Dispatch triggered allocating sub-pipelines: + +```mermaid +graph TD + A[Browser Request: /feat-x/api/health] --> B[Gateway: server.go] + B --> C{Proxy Cache lookup} + C -- Found --> D[Use Warm Proxy] + C -- Miss --> E[Supervisor: supervisor.go] + E --> F{Instance Started?} + F -- Yes --> G[Retrieve isolated dynamic Port] + F -- No --> H[Find Free Port findFreePort] + H --> I[Derive Instance Context context.WithCancel] + I --> J[Spawn Isolated LS & Sidecar] + J --> K[Save Port & Proc to instance Cache] + G --> L[Touch Trigger: Update LastAccessed] + K --> L + L --> M[Allocate proxyCache & Save] + D --> M + M --> N[httputil.ReverseProxy Dispatcher Dispatch] + + subgraph Reaper Daemon + O[Periodic Ticker: 1 min] --> P[Scan s.instances] + P --> Q{time.Since LastAccessed > 10m?} + Q -- Yes --> R[Call inst.cancel context abort] + R --> S[Reclaim ports/decr Supervisor process maps] + Q -- No --> T[No-op] + end +``` + +--- + +## 3. Detailed Proposed Changes + +The changes are grouped file by file across Go and Python workspaces. + +### 3.1 [Go-Gateway Component] + +#### [MODIFY] [gateway/supervisor.go](file:///topbar-frontend-switch/agent_manager/v2/gateway/supervisor.go) + +**A. Domain Object Remodelling** +Introduce a sub-structure grouping dynamic state to completely eliminate flat redundant locking maps pools tracking isolation: + +```go +// WorkspaceInstance maintains complete isolated lifecycle for one workspace. +type WorkspaceInstance struct { + Name string + Sidecar *managedProc + SidecarPort int + LS *managedProc + LSPort int + + ctx context.Context + cancel context.CancelFunc + lastAccessed time.Time +} +``` + +Update configuration structure arrays: +```go +type Supervisor struct { + // Binary sources for dynamic isolated allocation + SidecarBin string + LSBin string + + mu sync.RWMutex + cancel context.CancelFunc + ctx context.Context + + // The ALL-in-one mapping: + instances map[string]*WorkspaceInstance +} +``` + +**B. Refactoring Logic duplication helpers** +Refactor the redundant locking triggers residing in `GetOrStartSidecar` and `GetOrStartLanguageServer` into Unified generic allocatecycles mapping callers securely: + +```go +func (s *Supervisor) getOrStartProc( + workspaceName string, + serviceName string, + defaultPort int, + getProc func(*WorkspaceInstance) *managedProc, + setProc func(*WorkspaceInstance, *managedProc, int), + buildCmd func(port int) []string, +) int +``` + +**C. Touch reset triggers** +Provide thread-safe hooks for caller propagation updates: +```go +func (s *Supervisor) TouchInstance(name string) { + s.mu.Lock() + defer s.mu.Unlock() + if inst, ok := s.instances[name]; ok { + inst.lastAccessed = time.Now() + } +} +``` + +**D. Introduce Idle Instance Reaper (Garbage Collection)** +Start background routines deriving safety bounds enforcing memory cleanup triggers: +```go +func (s *Supervisor) StartReaper(ctx context.Context, idleTimeout time.Duration) { + ticker := time.NewTicker(1 * time.Minute) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + s.mu.Lock() + for name, inst := range s.instances { + if time.Since(inst.lastAccessed) > idleTimeout { + log.Printf("supervisor: reaping idle isolated workspace %s", name) + inst.cancel() + delete(s.instances, name) + } + } + s.mu.Unlock() + } + } +} +``` + +--- + +#### [MODIFY] [gateway/server.go](file:///topbar-frontend-switch/agent_manager/v2/gateway/server.go) + +**A. Proxy Caching** +Create locking concurrency pool preventing GC loads hot cycles allocates triggers: + +```go +type proxyCache struct { + mu sync.RWMutex + proxies map[string]*workspaceProxy +} + +func (pc *proxyCache) Get(ws string) *workspaceProxy +func (pc *proxyCache) Set(ws string, p *workspaceProxy) +``` + +**B. Cleanup static route branches triggers unification** +Route standard `head` resolving trigger dynamically constructed inside Router entrypoints wrappers safely. +Update both lookup route AND absolute `serveFromReferer` fallback guarantees symmetrically aligning dynamic propagation: + +```go + scPort := sup.GetOrStartSidecar(entry) + lsPort := sup.GetOrStartLanguageServer(entry) + + sup.TouchInstance(entry.Name) // Keep alive trigger! + + proxy := cache.Get(entry.Name) + if proxy == nil { + proxy = newWorkspaceProxy(scPort, lsPort, segment) + cache.Set(entry.Name, proxy) + } +``` + +--- + +#### [MODIFY] [gateway/registry.go](file:///topbar-frontend-switch/agent_manager/v2/gateway/registry.go) + +Prune configuration definitions structs aligned with absolute Isolation guarantees fully: + +```diff + type WorkspaceEntry struct { + Name string `json:"name"` + Path string `json:"path"` +- SidecarPort int `json:"sidecar_port"` +- LSPort int `json:"ls_port"` + BundlePath string `json:"bundle_path"` + } +``` + +--- + +#### [MODIFY] [gateway/main.go](file:///topbar-frontend-switch/agent_manager/v2/gateway/main.go) + +Eliminate redundant launch flags completely deprecating allocation static buffers overheads budgets cleanly: +- Deprecate `--ls_port` +- Deprecate `--sidecar_port` + +--- + +### 3.2 [CLI-Workspace Component] + +#### [MODIFY] [cli/registry.py](file:///topbar-frontend-switch/agent_manager/v2/cli/registry.py) + +Prune port specifications from workspace metadata definition schemas dataclasses symmetric support: + +```diff + @dataclasses.dataclass(frozen=True) + class WorkspaceEntry: + name: str + path: str +- sidecar_port: int +- ls_port: int + bundle_path: str = "" +``` + +Update `from_dict` filters ignoring backwards compatible legacy formats seamlessly. + +--- + +#### [MODIFY] [agent_manager.py](file:///topbar-frontend-switch/agent_manager/v2/agent_manager.py) + +**A. Prune Builder Writes triggers** +Remove redundant defaults ports allocations appended during frontend compiles initialization saves: +```diff + entry = reg_mod.WorkspaceEntry( + name=name, + path=str(worktree), +- sidecar_port=_DEFAULT_SIDECAR_PORT, +- ls_port=_DEFAULT_LS_PORT, + bundle_path=str(dist), + ) +``` + +**B. Accurate Status metrics triggers** +Modify redundant print scripts tracking outdated flat port lists to cleanly render isolation absolute capability accurately correctly. + +--- + +## 4. Migration Guidelines - Sweeps configuration limits + +Execute automated deployment migration pipeline safety strip limits sweeps existing disk setup configurations triggers safely backwards-compatible alignment guarantees: + +```python +import pathlib, json + +def migrate_workspaces(): + dir_path = pathlib.Path.home() / ".agent_manager" / "workspaces" + for p in dir_path.glob("*.json"): + data = json.loads(p.read_text()) + # Pop static metadata buffers safely + data.pop("sidecar_port", None) + data.pop("ls_port", None) + p.write_text(json.dumps(data, indent=2)) + print(f"Successfully Migrated Config format: {p.name}") + +if __name__ == "__main__": + migrate_workspaces() +``` + +--- + +## 5. Verification & Testing framework + +Complete end-to-end triggers validating safely: + +### 5.1 Code Verification compiles safety +- Run Standard Go target triggers + +- Verify Python style standards conformance: + `./agent_manager.py test` + +### 5.2 Live Instance Isolated dynamic trigger safety +1. Deploy Gateway binary launching pipeline. +2. Trigger isolated concurrency hits across parallel workspace environments routing triggers buffers. +3. Verify Supervisor allocated isolated port high-ranges correctly. +4. Verify Reaper ticks Reclaims dynamically aborted context stacks accurately transparently alleviating resource leak payloads. diff --git a/dataclass_array/memory/design/2026-03-26-sidecar-database-abstraction-v3.md b/dataclass_array/memory/design/2026-03-26-sidecar-database-abstraction-v3.md new file mode 100644 index 0000000..6d61874 --- /dev/null +++ b/dataclass_array/memory/design/2026-03-26-sidecar-database-abstraction-v3.md @@ -0,0 +1,678 @@ +# Sidecar Database Abstraction + +> **Status**: Design draft — 2026-03-26 (rev 3) +> +> **Scope**: Go sidecar backend + frontend data flow. +> +> **Starting point**: fresh — no v1 migration. +> +> **First use-case**: TODO Manager plugin. + +-------------------------------------------------------------------------------- + +## 1. Problem Statement + +The v2 sidecar returns stub empty arrays. We need a real persistence layer. +Constraints: + +- **All workspaces share one PostgreSQL database.** Each workspace has its own + sidecar process, but all connect to the same Postgres. Concurrency is a + first-class concern. +- **`Conversation` is a sidecar-owned core concept.** Any plugin can annotate + a conversation with plugin-specific metadata. No plugin owns or locks it. +- **Plugin data is self-contained.** Each plugin brings its own schema SQL and + HTTP routes. The sidecar core doesn't know the plugin internals. + +-------------------------------------------------------------------------------- + +## 2. Goals and Non-Goals + +**Goals** + +- `Conversation` is a core, generic, plugin-agnostic entity. +- Multiple plugins can independently annotate the same conversation. +- The response for a conversation merges all plugin annotations into a + `plugins` map: `{"todo-manager": {...}, "other-plugin": {...}}`. +- Clean plugin registration: each plugin contributes schema SQL + HTTP routes + + a `GetConversationMeta` hook. +- No wrapping of `pgx` — pass `*pgxpool.Pool` directly. +- Idempotent embedded SQL schema, advisory-lock protected. +- Gateway owns Postgres lifecycle (it already owns sidecar and LS). + +**Non-Goals** + +- No `Database` interface wrapping `pgxpool.Pool`. +- No ORM, no generic CRUD base. +- No real-time push — polling is fine. +- No v1 data import. + +-------------------------------------------------------------------------------- + +## 3. Architecture Overview + +``` +┌── Browser (React) ────────────────────────────────────────────────────────┐ +│ fetch('/api/conversations') → list + merged plugin metadata │ +│ fetch('/api/conversations/:id') → single with full plugin map │ +│ fetch('/api/todo-manager/todos') → plugin-specific endpoint │ +└────────────────────────────────────────────────────────────────────────────┘ + │ HTTP absolute path — Gateway routes by request metadata +┌── Gateway ────────────────────────────────────────────────────────────────┐ +│ Owns: sidecar lifecycle, LS lifecycle, Postgres lifecycle │ +│ Routes /head/api/* → sidecar :3000 │ +└───────────────────┬────────────────────────────────────────────────────────┘ + │ proxy (prefix stripped) +┌── Sidecar ─────────────────────────────────────────────────────────────────┐ +│ main.go │ +│ ├── db.Open(dsn) → *pgxpool.Pool │ +│ └── plugin.Registry → registers plugins, applies schema, wires routes │ +│ │ │ +│ ├── /api/health → healthHandler │ +│ ├── /api/conversations → ConversationHandler(registry) │ ← core +│ ├── /api/conversations/:id → ConversationHandler(registry) │ ← core +│ └── /api/todo-manager/todos/* → TodoManagerHandler(pool) │ ← plugin +│ │ +└────────────────────────────┬────────────────────────────────────────────────┘ + │ pgx pool +┌── PostgreSQL (shared) ──────▼────────────────────────────────────────────────┐ +│ conversations (sidecar core) │ +│ todo_manager_todos (todo-manager plugin) │ +│ todo_manager_conversation_meta (todo-manager plugin) │ +└──────────────────────────────────────────────────────────────────────────────┘ + ▲ ▲ ▲ + workspace A workspace B workspace C + (sidecar proc) (sidecar proc) (sidecar proc) +``` + +-------------------------------------------------------------------------------- + +## 4. Database Package — No `pgx` Wrapper + +`*pgxpool.Pool` is the concrete type used everywhere. No wrapper struct, no +interface. + +```go +// sidecar/db/db.go +package db + +// Open creates a connection pool and applies the embedded core schema. +// Also calls plugin.ApplySchema for each registered plugin. +// The advisory lock inside applySchema makes this safe for concurrent callers. +func Open(ctx context.Context, dsn string, plugins []plugin.Plugin) (*pgxpool.Pool, error) { + pool, err := pgxpool.New(ctx, dsn) + if err != nil { + return nil, fmt.Errorf("open pool: %w", err) + } + if err := ApplySchema(ctx, pool, plugins); err != nil { + pool.Close() + return nil, fmt.Errorf("apply schema: %w", err) + } + return pool, nil +} +``` + +Cross-cutting concerns (logging, tracing) belong in a `pgx.QueryTracer` set on +the pool config — pgx's own extension point. + +-------------------------------------------------------------------------------- + +## 5. Schema Strategy + +Single idempotent `CREATE TABLE IF NOT EXISTS` script — no migration library. +We're starting fresh; versioned migrations pay off when you need to ALTER a live +schema without a drop, which we don't have yet. + +Advisory lock during schema application ensures multiple sidecars racing at +startup don't interleave DDL: + +```go +// sidecar/db/schema.go +func ApplySchema(ctx context.Context, pool *pgxpool.Pool, plugins []plugin.Plugin) error { + conn, err := pool.Acquire(ctx) + if err != nil { return err } + defer conn.Release() + + const lockKey = int64(0x616d736368656d61) // "amschema" + if _, err := conn.Exec(ctx, "SELECT pg_advisory_lock($1)", lockKey); err != nil { + return fmt.Errorf("advisory lock: %w", err) + } + defer conn.Exec(ctx, "SELECT pg_advisory_unlock($1)", lockKey) //nolint:errcheck + + // Core schema (conversations table). + if _, err := conn.Exec(ctx, coreSchemaSQL); err != nil { + return fmt.Errorf("core schema: %w", err) + } + // Each plugin's schema. + for _, p := range plugins { + if sql := p.SchemaSQL(); sql != "" { + if _, err := conn.Exec(ctx, sql); err != nil { + return fmt.Errorf("plugin %s schema: %w", p.ID(), err) + } + } + } + return nil +} +``` + +-------------------------------------------------------------------------------- + +## 6. Plugin Interface + +Each plugin is a self-contained unit that contributes: + +1. **Schema SQL** — its own tables (embedded, idempotent). +2. **HTTP routes** — registered on the sidecar's mux. +3. **Conversation metadata hook** — queried when serving `GET + /api/conversations/:id`. + +```go +// sidecar/plugin/plugin.go +package plugin + +import ( + "context" + "net/http" + + "github.com/jackc/pgx/v5/pgxpool" +) + +// Plugin is implemented by each sidecar plugin. +type Plugin interface { + // ID returns the plugin's unique identifier (e.g. "todo-manager"). + ID() string + + // SchemaSQL returns idempotent DDL SQL for this plugin's tables. + // Called once at startup under the schema advisory lock. + SchemaSQL() string + + // RegisterRoutes wires this plugin's HTTP handlers onto mux. + // Called once at startup after schema is applied. + RegisterRoutes(mux *http.ServeMux) + + // GetConversationMeta returns the plugin's annotation for a given + // conversation ID, or nil if this plugin has no data for it. + // Called on every GET /api/conversations/:id. + GetConversationMeta(ctx context.Context, conversationID string) (any, error) +} + +// Registry holds all registered plugins and provides batch operations. +type Registry struct { + plugins []Plugin +} + +func NewRegistry(plugins ...Plugin) *Registry { + return &Registry{plugins: plugins} +} + +func (r *Registry) All() []Plugin { return r.plugins } + +// GetAllConversationMeta calls each plugin's GetConversationMeta and returns +// the merged map. Plugins with nil results are omitted. +func (r *Registry) GetAllConversationMeta(ctx context.Context, conversationID string) (map[string]any, error) { + result := make(map[string]any) + for _, p := range r.plugins { + meta, err := p.GetConversationMeta(ctx, conversationID) + if err != nil { + return nil, fmt.Errorf("plugin %s: %w", p.ID(), err) + } + if meta != nil { + result[p.ID()] = meta + } + } + return result, nil +} +``` + +-------------------------------------------------------------------------------- + +## 7. Core: `Conversation` + +A `Conversation` is identified directly by the **Antigravity cascade ID** — the same +ID that appears in the URL as `/c/`. There is no separate sidecar-assigned +UUID: the Antigravity ID is the primary key. + +### 7.1 Go type + +```go +// sidecar/store/conversation.go +package store + +import "time" + +// Conversation is the sidecar's core record for any Antigravity conversation +// launched from any plugin. Plugin-specific data lives in extension tables +// that reference this ID. +// +// ID and the Antigravity cascade ID are unified — the Antigravity ID is the PK. +type Conversation struct { + ID string // Antigravity cascade ID (/c/) + WorkspaceID string // which workspace launched this + CreatedAt time.Time +} + +// ConversationWithPlugins is the full response object served to the frontend. +type ConversationWithPlugins struct { + Conversation + Plugins map[string]any `json:"plugins"` // keyed by plugin ID +} +``` + +`Mode` (`navigate` / `direct`) is **not stored** — it is transient UI state used +only at conversation-creation time to decide how to open it in the frontend. +Once the conversation exists, it doesn't matter. + +### 7.2 Core schema + +```sql +-- sidecar/db/schema.sql (core only — embedded in db package) + +CREATE TABLE IF NOT EXISTS conversations ( + id TEXT PRIMARY KEY, -- Antigravity cascade ID + workspace_id TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE INDEX IF NOT EXISTS conversations_workspace_idx ON conversations(workspace_id); +``` + +### 7.3 Repository + +```go +// sidecar/store/conversation_repo.go +package store + +type ConversationRepo struct{ pool *pgxpool.Pool } + +func NewConversationRepo(pool *pgxpool.Pool) *ConversationRepo { + return &ConversationRepo{pool: pool} +} + +func (r *ConversationRepo) Create(ctx context.Context, id, workspaceID string) (Conversation, error) { + var c Conversation + err := r.pool.QueryRow(ctx, + `INSERT INTO conversations (id, workspace_id) + VALUES ($1, $2) + ON CONFLICT (id) DO NOTHING + RETURNING id, workspace_id, created_at`, + id, workspaceID, + ).Scan(&c.ID, &c.WorkspaceID, &c.CreatedAt) + // ON CONFLICT handles the race where two sidecars register the same cascade ID. + if err != nil { + return Conversation{}, fmt.Errorf("insert conversation %s: %w", id, err) + } + return c, nil +} + +func (r *ConversationRepo) ListAll(ctx context.Context) ([]Conversation, error) { + rows, err := r.pool.Query(ctx, + `SELECT id, workspace_id, created_at FROM conversations ORDER BY created_at DESC`) + if err != nil { + return nil, fmt.Errorf("list conversations: %w", err) + } + defer rows.Close() + return pgx.CollectRows(rows, pgx.RowToStructByName[Conversation]) +} + +func (r *ConversationRepo) Get(ctx context.Context, id string) (Conversation, error) { + var c Conversation + err := r.pool.QueryRow(ctx, + `SELECT id, workspace_id, created_at FROM conversations WHERE id=$1`, id, + ).Scan(&c.ID, &c.WorkspaceID, &c.CreatedAt) + if errors.Is(err, pgx.ErrNoRows) { + return Conversation{}, ErrNotFound + } + return c, err +} +``` + +### 7.4 Core HTTP handler + +``` +POST /api/conversations → create conversation record +GET /api/conversations → list all conversations +GET /api/conversations/:id → single conversation + merged plugin annotations +``` + +The list endpoint returns `ConversationWithPlugins` for each conversation. For +efficiency, plugins are queried **in batch per plugin**, not per conversation +(avoids N+1): + +```go +// GET /api/conversations +func (h *conversationHandler) handleList(w http.ResponseWriter, r *http.Request) { + convos, _ := h.repo.ListAll(ctx) + + // Batch: for each plugin, fetch all meta in one query. + pluginMetas := make(map[string]map[string]any) // pluginID → conversationID → meta + for _, p := range h.registry.All() { + metas, _ := p.BatchGetConversationMeta(ctx, convIDs(convos)) + pluginMetas[p.ID()] = metas + } + + // Assemble response. + var result []ConversationWithPlugins + for _, c := range convos { + plugins := map[string]any{} + for pluginID, metas := range pluginMetas { + if m, ok := metas[c.ID]; ok { + plugins[pluginID] = m + } + } + result = append(result, ConversationWithPlugins{Conversation: c, Plugins: plugins}) + } + writeJSON(w, http.StatusOK, result) +} +``` + +This adds a second method to the `Plugin` interface: + +```go +// BatchGetConversationMeta returns metadata for multiple conversations at once. +// Returns a map of conversationID → metadata. Missing IDs are omitted. +BatchGetConversationMeta(ctx context.Context, ids []string) (map[string]any, error) +``` + +-------------------------------------------------------------------------------- + +## 8. Plugin-Specific Data: Extension Tables + +Each plugin creates its own tables. Schema SQL lives **in the plugin folder**. +No shared metadata blob — typed extension tables are safer and queryable. + +### 8.1 TODO Manager schema + +```sql +-- sidecar/todo_manager/schema.sql (embedded in todo_manager package) + +CREATE TYPE IF NOT EXISTS todo_status AS ENUM ( + 'not_started', + 'in_progress', + 'needs_review', + 'done' +); + +CREATE TABLE IF NOT EXISTS todo_manager_todos ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + title TEXT NOT NULL, + parent_id UUID REFERENCES todo_manager_todos(id) ON DELETE CASCADE, + status todo_status NOT NULL DEFAULT 'not_started', + content TEXT NOT NULL DEFAULT '', + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- A todo can have many conversations (multiple attempts). +-- A conversation can be annotated by this plugin at most once. +CREATE TABLE IF NOT EXISTS todo_manager_conversation_meta ( + conversation_id TEXT PRIMARY KEY REFERENCES conversations(id) ON DELETE CASCADE, + todo_id UUID NOT NULL REFERENCES todo_manager_todos(id) ON DELETE CASCADE +); + +CREATE INDEX IF NOT EXISTS todo_meta_todo_idx ON todo_manager_conversation_meta(todo_id); +``` + +Key decisions: - `status` is a PostgreSQL `ENUM` — invalid values are rejected +at the DB level. - One conversation → at most one todo (via PK on +`conversation_id`). - One todo → many conversations (no unique constraint on +`todo_id`). + +### 8.2 TODO Manager Go types + +```go +// sidecar/todo_manager/types.go +package todomanager + +import "time" + +type Todo struct { + ID string `db:"id"` + Title string `db:"title"` + ParentID *string `db:"parent_id"` + Status string `db:"status"` + Content string `db:"content"` + CreatedAt time.Time `db:"created_at"` + UpdatedAt time.Time `db:"updated_at"` + Children []Todo `db:"-"` // assembled in-process from flat list +} + +// ConversationMeta is the plugin's annotation for a single conversation. +// This is what appears under "todo-manager" in the plugins map. +type ConversationMeta struct { + TodoID string `json:"todoId"` + Todo *Todo `json:"todo,omitempty"` // populated on request +} +``` + +### 8.3 TODO Manager Plugin implementation + +```go +// sidecar/todo_manager/plugin.go +package todomanager + +import ( + _ "embed" + "sidecar/plugin" +) + +//go:embed schema.sql +var schemaSQL string + +type TodoManagerPlugin struct { + repo *Repo +} + +func New(pool *pgxpool.Pool) *TodoManagerPlugin { + return &TodoManagerPlugin{repo: NewRepo(pool)} +} + +func (p *TodoManagerPlugin) ID() string { return "todo-manager" } +func (p *TodoManagerPlugin) SchemaSQL() string { return schemaSQL } + +func (p *TodoManagerPlugin) RegisterRoutes(mux *http.ServeMux) { + h := newHandler(p.repo) + mux.HandleFunc("/api/todo-manager/todos", h.handleTodos) + mux.HandleFunc("/api/todo-manager/todos/", h.handleTodo) +} + +func (p *TodoManagerPlugin) GetConversationMeta(ctx context.Context, id string) (any, error) { + meta, err := p.repo.GetConversationMeta(ctx, id) + if errors.Is(err, store.ErrNotFound) { + return nil, nil // this plugin has no data for this conversation + } + return meta, err +} + +func (p *TodoManagerPlugin) BatchGetConversationMeta(ctx context.Context, ids []string) (map[string]any, error) { + metas, err := p.repo.BatchGetConversationMeta(ctx, ids) + if err != nil { + return nil, err + } + result := make(map[string]any, len(metas)) + for k, v := range metas { + result[k] = v + } + return result, nil +} +``` + +-------------------------------------------------------------------------------- + +## 9. Concurrency: Multiple Sidecars, One Database + +| Situation | Mechanism | +| --------------------------------- | ---------------------------------------- | +| Schema DDL at startup (N sidecars | `pg_advisory_lock("amschema")` — only | +: racing) : one runs DDL, others wait and then no-op : +: : on `IF NOT EXISTS` : +| Inserting a new conversation | `ON CONFLICT (id) DO NOTHING` — two | +: : sidecars registering the same Antigravity ID : +: : is safe : +| Creating a todo | Plain `INSERT` — no two sidecars create | +: : the same todo : +| Updating a todo (optimistic) | `UPDATE ... WHERE id=$1 AND | +: : updated_at=$2` — returns 0 rows if : +: : stale; caller gets 409 : +| `status` validity | `ENUM` constraint enforced at DB level — | +: : no invalid status can be written : +| Read-then-mutate (e.g. reparent | `SELECT ... FOR UPDATE` inside explicit | +: subtree) : `BEGIN`/`COMMIT` : + +No `sync.Mutex` in repositories — that was the v1 workaround that only helped +within one process. + +-------------------------------------------------------------------------------- + +## 10. Postgres Lifecycle + +**The Gateway owns Postgres.** It already supervises the sidecar and Language +Server; Postgres is another supervised child in the same pattern. + +``` +Gateway startup: + 1. Start Postgres (or attach to an already-running instance) + 2. Wait for Postgres to accept connections + 3. Start sidecar (sidecar calls db.Open → pool + schema) + 4. Start LS +``` + +The DSN (Unix socket path or TCP address) is passed to the sidecar via a +`--db_dsn` flag, just as `--port` is passed today. The sidecar does not manage +Postgres — it only opens a pool to an already-running instance. + +-------------------------------------------------------------------------------- + +## 11. Frontend Data Flow + +### URL structure + +``` +/api/conversations ← sidecar core (generic, all plugins) +/api/conversations/:id ← single conversation + plugin map +/api/todo-manager/todos/* ← todo-manager plugin (todos only) +``` + +### Absolute paths + +Frontend code uses absolute paths (`fetch('/api/conversations')`). The gateway +routes requests to the correct sidecar using request metadata (e.g. Referer +header or session cookie that identifies the workspace). No workspace prefix +embedded in the fetch URL. + +> **Open design question**: the exact mechanism by which the gateway identifies +> the workspace from an absolute `/api/` request needs to be specified in the +> gateway design doc. This is intentionally deferred. + +### Response shape + +```ts +// GET /api/conversations +type ConversationList = ConversationWithPlugins[]; + +// GET /api/conversations/:id +interface ConversationWithPlugins { + id: string; // Antigravity cascade ID + workspaceId: string; + createdAt: string; // ISO 8601 + plugins: { + "todo-manager"?: { + todoId: string; + todo?: { id: string; title: string; status: string; ... }; + }; + // other plugins... + }; +} +``` + +### How the React store uses this + +```ts +// todo_manager/store.ts + +async function fetch(opts?: { force?: boolean }) { + const [todosResp, convoResp] = await Promise.all([ + window.fetch('/api/todo-manager/todos'), + window.fetch('/api/conversations'), + ]); + const todos: Todo[] = await todosResp.json(); + const conversations: ConversationWithPlugins[] = await convoResp.json(); + + // Extract this plugin's conversations from the shared list. + const myConvos = conversations.filter(c => 'todo-manager' in c.plugins); + // ... +} +``` + +### Creating a conversation + +1. Frontend calls `POST /api/conversations` with `{id: cascadeId, + workspaceId}`. +2. Sidecar core inserts into `conversations`. +3. Frontend (or plugin code) calls `POST /api/todo-manager/conversations` to + attach the extension data: `{conversationId, todoId}`. + +These are **two separate calls** — the sidecar core doesn't need to know about +the plugin's extension at creation time. If atomicity matters, the plugin +handler wraps both writes in a transaction. + +-------------------------------------------------------------------------------- + +## 12. File Layout + +``` +sidecar/ + main.go # Boot: Open pool → plugin.Registry → register routes + db/ + db.go # Open(ctx, dsn, plugins) *pgxpool.Pool + schema.go # ApplySchema (advisory lock) + schema.sql # Core schema: conversations table only + store/ + conversation.go # Conversation, ConversationWithPlugins types + conversation_repo.go # ConversationRepo + errors.go # ErrNotFound etc. + plugin/ + plugin.go # Plugin interface, Registry + todo_manager/ + schema.sql # todo_manager_todos + _conversation_meta + types.go # Todo, ConversationMeta + repo.go # TodoRepo + conv meta queries + handler.go # HTTP handlers + plugin.go # Implements plugin.Plugin + BUILD +``` + +-------------------------------------------------------------------------------- + +## 13. What We Are NOT Building + +| Omitted | Why | +| ----------------------------- | ------------------------------------------- | +| `Database` interface wrapping | No backend swap; the interface costs more | +: `pgxpool.Pool` : than it gives : +| ORM (sqlx, gorm, ent) | `pgx.CollectRows` + `pgx.RowToStructByName` | +: : is sufficient : +| `metadata JSONB` blob on | Typed extension tables are safer and | +: conversations : queryable : +| Per workspace database | Defeats the point — concurrency handled by | +: : PostgreSQL : +| Migration library | Idempotent `CREATE IF NOT EXISTS` is enough | +: : while schema is young : +| SSE / push | Polling is fine for the data size and | +: : refresh cadence : +| `mode` field on conversations | Transient UI concern — not a persistence | +: : concern : + +-------------------------------------------------------------------------------- + +## 14. Still Open + +| Question | Status | +| ---------------------------------- | --------------------------------------- | +| How does the gateway identify the | Needs a gateway design doc — options: | +: workspace from an absolute `/api/` : `Referer` header parsing, session : +: request? : cookie, `X-Workspace` header injected : +: : by the gateway before proxying : +| `BatchGetConversationMeta` returns | Fine for now; can add generics later if | +: `map[string]any` — should it be : needed : +: typed? : : +| Pagination on `GET | Add `?limit=&offset=` when the list | +: /api/conversations` : grows beyond a few hundred : diff --git a/dataclass_array/memory/design/2026-03-26-sidecar-database-abstraction.md b/dataclass_array/memory/design/2026-03-26-sidecar-database-abstraction.md new file mode 100644 index 0000000..14c6293 --- /dev/null +++ b/dataclass_array/memory/design/2026-03-26-sidecar-database-abstraction.md @@ -0,0 +1,590 @@ +# Sidecar Database Abstraction + +> **Status**: Design draft — 2026-03-26 (rev 2) +> +> **Scope**: Go sidecar backend + frontend data flow. +> +> **Starting point**: fresh — no migration from v1 JSON files. +> +> **First use-case**: TODO Manager plugin. + +-------------------------------------------------------------------------------- + +## 1. Problem Statement + +The v2 sidecar currently returns stub empty arrays. We need a real persistence +layer. The design constraints are: + +- **Multiple workspaces share one PostgreSQL database.** Each workspace has + its own sidecar process, but they all connect to the same Postgres instance. + Concurrency is a first-class concern, not an afterthought. +- **Multiple plugins** will accumulate data over time. The storage abstraction + must be clean enough to not become a mess as plugins multiply. +- **Starting from scratch** — no v1 compatibility required. + +-------------------------------------------------------------------------------- + +## 2. Goals and Non-Goals + +**Goals** + +- Single shared PostgreSQL database for all workspaces. +- Generic `Conversation` type owned by the sidecar core (not by any one + plugin). Plugins attach plugin-specific data via extension tables. +- Clean per-plugin repository layer — no SQL in HTTP handlers. +- No wrapping of `pgx` — use `*pgxpool.Pool` directly. +- Schema applied via embedded SQL at startup, with advisory-lock concurrency + safety so multiple sidecars racing at boot are safe. +- Workspace-scoped reads where appropriate — queries filter by `workspace_id`. +- Frontend data access is explicit: REST endpoints, registered per plugin. + +**Non-Goals** + +- No interface wrapping `pgxpool.Pool`. +- No ORM, no generic CRUD base. +- No real-time push (SSE/WebSocket) in v1 — polling is fine. +- No v1 data import. + +-------------------------------------------------------------------------------- + +## 3. Architecture Overview + +``` +┌── Browser (React) ──────────────────────────────────────────────────────┐ +│ TodoStore: fetch("/api/todo-manager/todos") │ +│ fetch("/api/todo-manager/conversations") │ +└─────────────────────────────────────────────────────────────────────────┘ + │ HTTP (via Gateway prefix routing) +┌── Sidecar (Go) ──────────────────────────────────────────────────────────┐ +│ │ +│ main.go │ +│ └─ db.Open(dsn) → *pgxpool.Pool (shared across all handlers) │ +│ └─ registerRoutes(mux, pool) │ +│ │ │ +│ ├── /api/health → healthHandler │ +│ ├── /api/conversations/* → ConversationHandler(pool) │ ← sidecar core +│ └── /api/todo-manager/* → TodoManagerHandler(pool) │ ← plugin +│ │ +└──────────────────────────────────────┬───────────────────────────────────┘ + │ pgx pool (multiple connections) +┌── PostgreSQL (shared) ───────────────▼───────────────────────────────────┐ +│ conversations (sidecar-owned, generic) │ +│ todo_manager_todos (plugin-owned) │ +│ todo_manager_conversation_meta (join: conversations ↔ todo) │ +└──────────────────────────────────────────────────────────────────────────┘ + ▲ ▲ ▲ + workspace A workspace B workspace C + (sidecar proc) (sidecar proc) (sidecar proc) +``` + +-------------------------------------------------------------------------------- + +## 4. Do We Wrap `pgx`? + +**Decision: No.** + +`*pgxpool.Pool` is the concrete type threaded through the entire application. No +`Database` interface, no wrapper struct. + +**Rationale:** + +- A `Database` interface over `pgx` requires re-implementing every primitive + (`QueryRow`, `Exec`, `BeginTx`, batch...). Every new pgx feature forces a + new interface method. +- We will never swap Postgres for another backend — the interface buys + nothing. +- Cross-cutting concerns (logging, tracing) belong in a `pgx.QueryTracer` + attached to the pool config — pgx's own extension point. +- Tests use a real Postgres connection (via `pgxmock` or a test container). + +The `db` package is a thin boot helper: + +```go +// sidecar/db/db.go +package db + +import ( + "context" + "fmt" + + "github.com/jackc/pgx/v5/pgxpool" +) + +// Open creates a connection pool and applies the embedded schema. +// The advisory lock inside applySchema makes this safe for concurrent callers. +func Open(ctx context.Context, dsn string) (*pgxpool.Pool, error) { + pool, err := pgxpool.New(ctx, dsn) + if err != nil { + return nil, fmt.Errorf("open pool: %w", err) + } + if err := applySchema(ctx, pool); err != nil { + pool.Close() + return nil, fmt.Errorf("apply schema: %w", err) + } + return pool, nil +} +``` + +-------------------------------------------------------------------------------- + +## 5. Schema Strategy: No Migration Library + +Since we're starting fresh and the schema will evolve rapidly, we use a **single +idempotent `CREATE TABLE IF NOT EXISTS` script** embedded in the binary, rather +than a versioned migration library. A migration library adds complexity (version +table, up/down files) that pays off when you need to evolve an existing +production schema — which we don't have yet. + +When the schema genuinely needs an incompatible change, we drop and recreate the +database (during development) or add a new version field and handle it ad hoc. + +```go +// sidecar/db/schema.go +package db + +import ( + _ "embed" + "context" + "fmt" + + "github.com/jackc/pgx/v5/pgxpool" +) + +//go:embed schema.sql +var schemaSQL string + +// applySchema applies the schema idempotently. +// Uses a Postgres advisory lock so multiple sidecars racing at startup +// do not interleave DDL statements. +func applySchema(ctx context.Context, pool *pgxpool.Pool) error { + conn, err := pool.Acquire(ctx) + if err != nil { + return fmt.Errorf("acquire conn for schema: %w", err) + } + defer conn.Release() + + // Advisory lock: all sidecars share lock key 0x616d /* 'am' */ schema. + const lockKey = 0x616d736368656d61 // "amschema" + if _, err := conn.Exec(ctx, "SELECT pg_advisory_lock($1)", lockKey); err != nil { + return fmt.Errorf("advisory lock: %w", err) + } + defer conn.Exec(ctx, "SELECT pg_advisory_unlock($1)", lockKey) //nolint:errcheck + + if _, err := conn.Exec(ctx, schemaSQL); err != nil { + return fmt.Errorf("apply schema.sql: %w", err) + } + return nil +} +``` + +`schema.sql` is a single file with `IF NOT EXISTS` guards throughout — running +it multiple times is safe. + +-------------------------------------------------------------------------------- + +## 6. Generic Conversation (Sidecar Core) + +A `Conversation` is a record that any plugin can create. It is the **sidecar's +own concept** — not tied to any one plugin's domain. Plugins attach their own +data by referencing the conversation ID in plugin-specific extension tables. + +### 6.1 Go type + +```go +// sidecar/store/conversation.go +package store + +import "time" + +// Conversation represents a Antigravity conversation launched from any plugin. +// It is the sidecar-owned generic record; plugin-specific data lives in +// extension tables that reference this ID. +type Conversation struct { + ID string // UUID — primary key + WorkspaceID string // which workspace created this conversation + PluginID string // which plugin owns it (e.g. "todo-manager") + ConversationID string // Antigravity cascade ID (appears in URL as /c/) + LaunchedAt time.Time + Mode string // plugin-defined; e.g. "navigate" | "direct" +} +``` + +### 6.2 Repository + +```go +// sidecar/store/conversation_repo.go +package store + +import ( + "context" + "fmt" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" +) + +type ConversationRepo struct { + pool *pgxpool.Pool +} + +func NewConversationRepo(pool *pgxpool.Pool) *ConversationRepo { + return &ConversationRepo{pool: pool} +} + +// Create inserts a new conversation and returns it with its assigned ID. +func (r *ConversationRepo) Create(ctx context.Context, workspaceID, pluginID, conversationID, mode string) (Conversation, error) { + var c Conversation + err := r.pool.QueryRow(ctx, ` + INSERT INTO conversations (workspace_id, plugin_id, conversation_id, mode) + VALUES ($1, $2, $3, $4) + RETURNING id, workspace_id, plugin_id, conversation_id, launched_at, mode`, + workspaceID, pluginID, conversationID, mode, + ).Scan(&c.ID, &c.WorkspaceID, &c.PluginID, &c.ConversationID, &c.LaunchedAt, &c.Mode) + if err != nil { + return Conversation{}, fmt.Errorf("insert conversation: %w", err) + } + return c, nil +} + +// ListByPlugin returns all conversations for a given plugin, across all workspaces. +func (r *ConversationRepo) ListByPlugin(ctx context.Context, pluginID string) ([]Conversation, error) { + rows, err := r.pool.Query(ctx, ` + SELECT id, workspace_id, plugin_id, conversation_id, launched_at, mode + FROM conversations + WHERE plugin_id = $1 + ORDER BY launched_at DESC`, + pluginID) + if err != nil { + return nil, fmt.Errorf("list conversations: %w", err) + } + defer rows.Close() + return pgx.CollectRows(rows, pgx.RowToStructByName[Conversation]) +} +``` + +### 6.3 HTTP handler + +The sidecar exposes a generic `/api/conversations/` endpoint so the frontend can +create and query conversations without knowing which plugin owns them. + +``` +GET /api/conversations?plugin=todo-manager → list conversations for plugin +POST /api/conversations → create (returns {id}) +``` + +-------------------------------------------------------------------------------- + +## 7. Plugin-Specific Data: Extension Tables + +Each plugin that needs to attach data to a conversation creates an **extension +table** with a FK to `conversations(id)`. No shared `metadata JSONB` blob — that +trades type safety for flexibility we don't need. + +### 7.1 TODO Manager example + +The `todo-manager` plugin links each conversation to a specific `Todo` and +records a `mode`: + +```sql +-- In schema.sql + +-- ── SIDECAR CORE ───────────────────────────────────────────────────────── + +CREATE TABLE IF NOT EXISTS conversations ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + workspace_id TEXT NOT NULL, + plugin_id TEXT NOT NULL, + conversation_id TEXT NOT NULL, + launched_at TIMESTAMPTZ NOT NULL DEFAULT now(), + mode TEXT NOT NULL DEFAULT '' +); + +CREATE INDEX IF NOT EXISTS conversations_plugin_idx ON conversations(plugin_id); +CREATE INDEX IF NOT EXISTS conversations_workspace_idx ON conversations(workspace_id); + +-- ── TODO MANAGER PLUGIN ─────────────────────────────────────────────────── + +CREATE TABLE IF NOT EXISTS todo_manager_todos ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + title TEXT NOT NULL, + parent_id UUID REFERENCES todo_manager_todos(id) ON DELETE CASCADE, + status TEXT NOT NULL DEFAULT 'not_started', + content TEXT NOT NULL DEFAULT '', + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- Extension table: attaches a todo to any sidecar conversation. +CREATE TABLE IF NOT EXISTS todo_manager_conversation_meta ( + conversation_id UUID PRIMARY KEY REFERENCES conversations(id) ON DELETE CASCADE, + todo_id UUID NOT NULL REFERENCES todo_manager_todos(id) ON DELETE CASCADE +); + +CREATE INDEX IF NOT EXISTS todo_conv_meta_todo_idx ON todo_manager_conversation_meta(todo_id); +``` + +### 7.2 Assembled Go type served to the frontend + +The plugin repository **joins** the generic conversation with its own extension +table, producing a rich type: + +```go +// sidecar/todo_manager/types.go +package todomanager + +import ( + "time" + "sidecar/store" +) + +// TodoConversation is the assembled view served to the frontend. +// It embeds the generic Conversation and adds todo-specific fields. +type TodoConversation struct { + store.Conversation // id, workspace_id, launched_at, mode, ... + TodoID string // from todo_manager_conversation_meta + Todo *Todo // optionally JOIN-populated +} + +type Todo struct { + ID string `db:"id"` + Title string `db:"title"` + ParentID *string `db:"parent_id"` + Status string `db:"status"` + Content string `db:"content"` + CreatedAt time.Time `db:"created_at"` + UpdatedAt time.Time `db:"updated_at"` + Children []Todo `db:"-"` // assembled in-process +} +``` + +The repository query: + +```go +// Joins conversations + meta + todos in one query. +func (r *Repo) ListConversations(ctx context.Context) ([]TodoConversation, error) { + rows, err := r.pool.Query(ctx, ` + SELECT + c.id, c.workspace_id, c.plugin_id, c.conversation_id, c.launched_at, c.mode, + m.todo_id, + t.title AS todo_title, t.status AS todo_status + FROM conversations c + JOIN todo_manager_conversation_meta m ON m.conversation_id = c.id + JOIN todo_manager_todos t ON t.id = m.todo_id + WHERE c.plugin_id = 'todo-manager' + ORDER BY c.launched_at DESC`) + ... +} +``` + +**Creating a TodoConversation** is one atomic transaction — it inserts into both +`conversations` and `todo_manager_conversation_meta`: + +```go +func (r *Repo) CreateConversation(ctx context.Context, req CreateConversationReq) (TodoConversation, error) { + tx, err := r.pool.Begin(ctx) + if err != nil { + return TodoConversation{}, err + } + defer tx.Rollback(ctx) + + var c store.Conversation + err = tx.QueryRow(ctx, ` + INSERT INTO conversations (workspace_id, plugin_id, conversation_id, mode) + VALUES ($1, 'todo-manager', $2, $3) + RETURNING id, workspace_id, plugin_id, conversation_id, launched_at, mode`, + req.WorkspaceID, req.ConversationID, req.Mode, + ).Scan(&c.ID, &c.WorkspaceID, &c.PluginID, &c.ConversationID, &c.LaunchedAt, &c.Mode) + if err != nil { + return TodoConversation{}, fmt.Errorf("insert conversation: %w", err) + } + + if _, err = tx.Exec(ctx, ` + INSERT INTO todo_manager_conversation_meta (conversation_id, todo_id) + VALUES ($1, $2)`, c.ID, req.TodoID); err != nil { + return TodoConversation{}, fmt.Errorf("insert conversation meta: %w", err) + } + + if err = tx.Commit(ctx); err != nil { + return TodoConversation{}, fmt.Errorf("commit: %w", err) + } + return TodoConversation{Conversation: c, TodoID: req.TodoID}, nil +} +``` + +-------------------------------------------------------------------------------- + +## 8. Concurrency — Multiple Sidecars, One Database + +**The hard constraint**: every workspace has its own sidecar process, and all +processes connect to the same Postgres. We can have N concurrent writers. + +### What PostgreSQL gives us for free + +- **Atomic INSERT/UPDATE/DELETE** — no application-level mutex needed for + individual writes. +- **SERIALIZABLE / READ COMMITTED** isolation — default `READ COMMITTED` is + sufficient for our use cases (no read-then-write invariants on the same row + from two clients simultaneously). +- **UNIQUE constraints** — enforce uniqueness at the DB level, not in the app. + +### Patterns we apply + +| Situation | Mechanism | +| --------------------------------- | ---------------------------------------- | +| Schema DDL at startup (multiple | `pg_advisory_lock` in `applySchema` — | +: sidecars) : only one executes DDL, others wait : +| Simple inserts (new todo, new | Plain `INSERT` — each sidecar writes its | +: conversation) : own rows independently : +| Update a todo (optimistic) | `UPDATE ... WHERE id=$1 AND | +: : updated_at=$2 RETURNING id` — returns 0 : +: : rows if stale; caller retries : +| Read-then-mutate (e.g. reparent a | `SELECT ... FOR UPDATE` inside an | +: subtree) : explicit `BEGIN`/`COMMIT` : +| Ensuring a conversation_id is not | `UNIQUE(conversation_id)` constraint on | +: duplicated : `conversations` table : + +### Workspace-scoped queries + +All sidecar processes share the same `conversations` table. Where data is +workspace-specific (e.g. "show me my conversations"), queries **filter by +`workspace_id`**. The `workspace_id` is injected by the sidecar at startup from +the `--workspace` flag the gateway already passes. + +Todos, however, are **shared across all workspaces** — the todo tree is a global +data model, not per-workspace. Only the conversations that link to those todos +are workspace-scoped. + +-------------------------------------------------------------------------------- + +## 9. Frontend Data Flow + +The React frontend (plugin code) calls the sidecar via relative `fetch()` calls +through the gateway. Each plugin owns its own URL namespace. + +### URL namespacing + +``` +/api/conversations ← sidecar core (generic) +/api/todo-manager/* ← todo-manager plugin +/api//* ← other future plugins +``` + +### Todo Manager endpoints + +| Method | Path | Description | +| -------- | --------------------------------- | -------------------- | +| `GET` | `/api/todo-manager/todos` | Returns full todo | +: : : tree : +| `POST` | `/api/todo-manager/todos` | Create a new todo | +| `PUT` | `/api/todo-manager/todos/:id` | Update | +: : : title/status/content : +| `DELETE` | `/api/todo-manager/todos/:id` | Delete (cascades to | +: : : conversation meta) : +| `GET` | `/api/todo-manager/conversations` | Returns | +: : : `[]TodoConversation` : +: : : (joined) : +| `POST` | `/api/todo-manager/conversations` | Create | +: : : conversation+meta : +: : : atomically : + +### Generic conversation endpoints + +Method | Path | Description +------ | -------------------------------- | ----------------------------------- +`GET` | `/api/conversations?plugin=` | List all conversations for a plugin + +### How the React store calls this + +```ts +// todo_manager/store.ts + +async function fetch(opts?: { force?: boolean }) { + const [todosResp, convoResp] = await Promise.all([ + window.fetch('api/todo-manager/todos'), + window.fetch('api/todo-manager/conversations'), + ]); + // ... +} + +async function registerConversation(entry: Omit) { + await window.fetch('api/todo-manager/conversations', { + method: 'POST', + headers: {'Content-Type': 'application/json'}, + body: JSON.stringify(entry), + }); +} +``` + +Note: `fetch('api/...')` not `fetch('/api/...')` — relative paths are required +by the gateway's root isolation middleware. The sidecar receives the path +already stripped of the workspace prefix. + +### Polling strategy + +The frontend polls on mount and on explicit user refresh. The sidecar does NOT +push — no SSE, no WebSocket. `ConversationStatus` (running/idle) is derived from +`useAgentStateProvider` (Antigravity's own stream), not from the database. + +-------------------------------------------------------------------------------- + +## 10. File Layout + +``` +sidecar/ + main.go # Startup: Open pool, register routes + db/ + db.go # Open(ctx, dsn) *pgxpool.Pool + schema.go # applySchema (advisory lock + embedded SQL) + schema.sql # Single idempotent CREATE TABLE IF NOT EXISTS file + store/ + conversation.go # Conversation type + conversation_repo.go # ConversationRepo (sidecar core) + todo_manager/ + types.go # Todo, TodoConversation, CreateConversationReq + todo_repo.go # TodoRepo (CRUD on todo_manager_todos) + conversation_repo.go # Repo.ListConversations, Repo.CreateConversation + handler.go # HTTP handlers — call repos, no SQL here + BUILD +``` + +-------------------------------------------------------------------------------- + +## 11. What We Are NOT Building + +| Omitted | Why | +| --------------------------------- | ---------------------------------------- | +| `Database` interface wrapping | No backend swap planned; interfacing pgx | +: `pgxpool.Pool` : costs more than it gives : +| ORM (sqlx, gorm, ent) | `pgx.CollectRows` + | +: : `pgx.RowToStructByName` is sufficient : +| Generic CRUD base struct | Plugin data models are distinct enough | +| `metadata JSONB` on conversations | Typed extension tables are safer and | +: : queryable : +| Per-workspace database isolation | Defeats the purpose of sharing state; | +: : concurrency via PostgreSQL : +| Migration library | Idempotent `CREATE IF NOT EXISTS` is | +: : enough while schema is young : +| SSE / push | Polling is fine for the current data | +: : size and refresh rate : + +-------------------------------------------------------------------------------- + +## 12. Open Questions + +| Question | Status | +| ----------------------------------- | -------------------------------------- | +| Where does the shared Postgres run? | To decide — could be system Postgres | +: (socket path, port, user) : on a fixed port, or a Unix socket at : +: : `~/.agent-manager/pgsql/.s.PGSQL.5432` : +: : owned by the user. : +| Who starts Postgres if it's not | The sidecar or a one-time CLI setup | +: already running? : command? Recommendation\: CLI : +: : (`agent_manager.py setup`) to keep : +: : sidecar startup simple. : +| `--workspace` flag format — how is | Gateway already passes it; verify flag | +: it passed today? : name in `main.go`. : +| Notion todos: stored in DB or | Fetched fresh from Notion API on every | +: fetched fresh? : `GET /api/todo-manager/todos` with TTL : +: : cache, NOT stored. Only native todos : +: : are DB-backed. : diff --git a/dataclass_array/memory/design/2026-03-26-sidecar-database-design.md b/dataclass_array/memory/design/2026-03-26-sidecar-database-design.md new file mode 100644 index 0000000..ce327d5 --- /dev/null +++ b/dataclass_array/memory/design/2026-03-26-sidecar-database-design.md @@ -0,0 +1,457 @@ +# Sidecar Database Abstraction Design + +**Date:** 2026-03-26 **Status:** Draft + +-------------------------------------------------------------------------------- + +## 1. Problem statement + +The v1 sidecar stored plugin data (e.g. `TodoConversation` records) in flat JSON +files (`~/.agent-manager/conversations.json`). This approach does not scale: + +- No concurrent write safety (two sidecar instances race on the same file). +- No indexing or querying — every read loads the entire file. +- Schema evolution requires hand-rolled migration code. + +The v2 sidecar will use **PostgreSQL** (via the +[pgx](https://github.com/jackc/pgx) Go driver) as its database. + +This document designs the layered abstraction so that: + +1. The sidecar manages one PostgreSQL connection pool. +2. Each plugin receives a domain-specific repository, not a raw DB handle. +3. SQL is encapsulated; plugin business logic never sees a query string. + +-------------------------------------------------------------------------------- + +## 2. Do we actually need these abstractions? + +### 2.1 Is a PostgreSQL wrapper layer necessary? + +**Short answer: No generic "DB abstraction" — but yes, a testable interface.** + +pgx already provides an excellent, idiomatic Go API. Wrapping it in a `Database` +interface that hides `pgxpool.Pool` would be pointless indirection: we are not +going to swap PostgreSQL for SQLite, and pgx is not a legacy API that needs +cleaning up. + +What *is* worth having is a thin interface over `pgxpool.Pool` so that tests can +inject a fake or a `pgxmock` without starting a real database. The standard Go +pattern is to accept an interface (`db.Querier`) rather than a concrete type. +pgx v5 ships its own `pgxpool.Pool` but exposes the `pgx.Tx` and `pgconn` +interfaces; we can carve out whatever subset we need. + +**Decision:** expose a `Querier` interface (`ExecContext`, `QueryRow`, +`QueryRows`) that `*pgxpool.Pool` and `*pgx.Tx` both satisfy. The sidecar +creates and owns the concrete pool. Plugins never see `pgxpool.Pool` — only the +`Querier` passed to their repository constructor. + +### 2.2 Is a per-plugin domain mapping layer necessary? + +**Yes — and for a principled reason, not ceremony.** + +Consider the alternative: plugin handlers call `pool.Query(ctx, "SELECT ... FROM +conversations WHERE todo_id = $1", todoId)` directly. Problems: + +- SQL leaks into HTTP handler business logic, mixing abstraction levels. +- The same query is copy–pasted across handlers. +- Tests must mock at the SQL level, not the domain level. +- Schema changes require hunting every query string in the codebase. + +The **repository pattern** solves this cleanly: each plugin defines a +`XxxRepository` interface with domain-level methods (`GetConversations(todoId +string) ([]TodoConversation, error)`), and one concrete implementation backed by +pgx. HTTP handlers operate on the interface; tests substitute a fake. + +This is not a generic ORM — there is no reflection, no `struct` tag scanning, no +`Model()` helper. Each repository is a small, hand-written adapter (~50–100 +lines) that translates between SQL rows and domain structs. + +-------------------------------------------------------------------------------- + +## 3. Architecture + +``` +sidecar main.go + │ + ├── pgxpool.Pool (created once at startup, closed at shutdown) + │ │ + │ └── wraps pgx connection pool to Postgres + │ + ├── db.Querier (interface, satisfied by *pgxpool.Pool and *pgx.Tx) + │ │ + │ ├── todo-manager plugin + │ │ └── todo.Repository (interface) + │ │ └── todo.pgxRepository (concrete, holds Querier) + │ │ SQL ↔ TodoConversation mapping + │ │ + │ └── future-plugin + │ └── future.Repository ... + │ + └── HTTP handlers (receive repository interfaces as dependencies) +``` + +### Dependency flow + +``` +main.go + pool := pgxpool.New(ctx, dsn) + q := db.NewQuerier(pool) // wraps pool, no new layer + repo := todo.NewRepository(q) // domain-level adapter + srv := todo.NewServer(repo) // HTTP handlers + mux.Handle("/api/conversations", srv) +``` + +No globals. Each dependency is passed explicitly down the call chain (api-design +principle: *pass dependencies, don't reach for globals*). + +-------------------------------------------------------------------------------- + +## 4. Layer specifications + +### 4.1 `sidecar/db` — Querier interface + +```go +// Package db defines the minimal database interface used by sidecar plugins. +// It is satisfied by *pgxpool.Pool, *pgx.Tx, and any test double. +package db + +import ( + "context" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgconn" +) + +// Querier is the subset of pgxpool.Pool that plugins need. +// Both *pgxpool.Pool and *pgx.Tx implement this interface. +type Querier interface { + Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error) + Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error) + QueryRow(ctx context.Context, sql string, args ...any) pgx.Row +} + +// WithTx runs fn inside a transaction, committing on success and rolling back +// on error or panic. pool must be *pgxpool.Pool. +func WithTx(ctx context.Context, pool interface { + Begin(context.Context) (pgx.Tx, error) +}, fn func(Querier) error) error { + tx, err := pool.Begin(ctx) + if err != nil { + return err + } + defer func() { + if p := recover(); p != nil { + _ = tx.Rollback(ctx) + panic(p) + } + }() + if err := fn(tx); err != nil { + _ = tx.Rollback(ctx) + return err + } + return tx.Commit(ctx) +} +``` + +**Why only three methods?** YAGNI. Add `CopyFrom`, `SendBatch`, etc. when a +plugin actually needs them. The interface stays narrow. + +**Why not pgxpool.Pool directly?** Tests would need a real PostgreSQL process. +With `Querier`, a test can pass a `pgxmock` or a simple in-memory fake. + +### 4.2 `sidecar/db/migrate` — Schema migrations + +Lightweight, no framework. Migrations are numbered SQL files embedded in the +binary via `go:embed`: + +``` +sidecar/db/migrate/ + 0001_create_conversations.sql + 0002_add_template_col.sql + ... +``` + +`migrate.Run(ctx, pool)` applies any unapplied migrations in order, using a +`schema_migrations` table as the applied-set tracker. This is the minimal +version of what Flyway / golang-migrate do, without the external dependency. + +```go +// migrate.AppliedMigrations returns the set of already-applied migration IDs. +// migrate.Apply runs a single migration in a transaction. +// migrate.Run applies all pending migrations on startup. +``` + +### 4.3 Plugin repository — example: `sidecar/todo` + +The TODO plugin needs to store `TodoConversation` records (see v2 research doc). + +```go +// Package todo implements the sidecar-side of the TODO Manager plugin. +package todo + +import ( + "context" + "time" + + ".../sidecar/db" +) + +// Conversation is the domain object (mirrors the frontend's TodoConversation). +type Conversation struct { + ID string // local UUID + TodoID string // Notion page UUID + ConversationID string // Jetbox cascade ID + LaunchedAt time.Time + Mode string // "navigate" | "direct" +} + +// Repository is the domain-level interface for conversation persistence. +// Handlers depend on this interface, not on the concrete pgx implementation. +type Repository interface { + ListByTodo(ctx context.Context, todoID string) ([]Conversation, error) + List(ctx context.Context) ([]Conversation, error) + Create(ctx context.Context, c Conversation) (Conversation, error) +} + +// NewRepository returns a Repository backed by q. +func NewRepository(q db.Querier) Repository { + return &pgxRepository{q: q} +} + +type pgxRepository struct{ q db.Querier } + +func (r *pgxRepository) List(ctx context.Context) ([]Conversation, error) { + rows, err := r.q.Query(ctx, ` + SELECT id, todo_id, conversation_id, launched_at, mode + FROM todo_conversations + ORDER BY launched_at DESC + `) + if err != nil { + return nil, err + } + defer rows.Close() + return scanConversations(rows) +} + +func (r *pgxRepository) ListByTodo(ctx context.Context, todoID string) ([]Conversation, error) { + rows, err := r.q.Query(ctx, ` + SELECT id, todo_id, conversation_id, launched_at, mode + FROM todo_conversations + WHERE todo_id = $1 + ORDER BY launched_at DESC + `, todoID) + if err != nil { + return nil, err + } + defer rows.Close() + return scanConversations(rows) +} + +func (r *pgxRepository) Create(ctx context.Context, c Conversation) (Conversation, error) { + c.ID = newUUID() + c.LaunchedAt = time.Now().UTC() + _, err := r.q.Exec(ctx, ` + INSERT INTO todo_conversations (id, todo_id, conversation_id, launched_at, mode) + VALUES ($1, $2, $3, $4, $5) + `, c.ID, c.TodoID, c.ConversationID, c.LaunchedAt, c.Mode) + if err != nil { + return Conversation{}, err + } + return c, nil +} +``` + +The corresponding migration: + +```sql +-- 0001_create_conversations.sql +CREATE TABLE IF NOT EXISTS todo_conversations ( + id TEXT PRIMARY KEY, + todo_id TEXT NOT NULL, + conversation_id TEXT NOT NULL, + launched_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + mode TEXT NOT NULL CHECK (mode IN ('navigate', 'direct')) +); + +CREATE INDEX IF NOT EXISTS idx_todo_conversations_todo_id + ON todo_conversations (todo_id); +``` + +### 4.4 HTTP handlers wired to the repository + +```go +// Server holds the HTTP handlers for the TODO plugin. +type Server struct{ repo Repository } + +func NewServer(repo Repository) *Server { return &Server{repo: repo} } + +func (s *Server) HandleList(w http.ResponseWriter, r *http.Request) { + convs, err := s.repo.List(r.Context()) + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + writeJSON(w, http.StatusOK, convs) +} + +func (s *Server) HandleCreate(w http.ResponseWriter, r *http.Request) { + var req struct { + TodoID string `json:"todoId"` + ConversationID string `json:"conversationId"` + Mode string `json:"mode"` + } + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + created, err := s.repo.Create(r.Context(), Conversation{ + TodoID: req.TodoID, + ConversationID: req.ConversationID, + Mode: req.Mode, + }) + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + writeJSON(w, http.StatusCreated, created) +} +``` + +-------------------------------------------------------------------------------- + +## 5. Database topology (one DB per workspace vs. shared) + +The "preventing database race conditions" research +([conversation dab8ea46](https://docs.google.com/...)) concluded that **each +workspace should have its own PostgreSQL database** (or schema). + +Rationale: + +- Sidecars for different workspaces run as separate processes. A shared DB + requires careful locking to avoid corruption on concurrent writes. +- A workspace-scoped DB is simpler: no cross-workspace JOIN ever makes sense; + dropping a workspace is a `DROP DATABASE`; connection-level isolation is + free. +- PostgreSQL schema-per-workspace (same server, different schema) is an + acceptable alternative if spinning up a DB per workspace is expensive. + +**Decision for v2:** one PostgreSQL **schema per workspace**, all sharing the +same server instance. The schema name is the workspace name (e.g., `head`, +`feat_x`). The DSN passed to `pgxpool.New` includes `search_path=`. + +This is enforced in `main.go`: + +```go +dsn := fmt.Sprintf( + "host=%s port=%d dbname=agent_manager user=%s password=%s search_path=%s sslmode=disable", + pgHost, pgPort, pgUser, pgPassword, sanitizeSchema(workspaceName), +) +pool, err := pgxpool.New(ctx, dsn) +``` + +Migrations run against `search_path=`, so each workspace gets its own +table set. + +-------------------------------------------------------------------------------- + +## 6. Testability + +| Level | Strategy | +| --------------------- | --------------------------------------------------- | +| Unit (repository) | Pass a `pgxmock` mock that satisfies `db.Querier` | +| Integration (handler) | `httptest.NewRecorder` + repository fake | +: : (`fakeRepository` struct satisfying the interface) : +| End-to-end | Real PostgreSQL via `testcontainers-go` or a shared | +: : local `postgres\:alpine` : + +The `Querier` interface is the key seam. Because handlers receive a `Repository` +interface (not the concrete pgx implementation), fake repositories are trivial +to write: + +```go +type fakeRepository struct{ convs []Conversation } +func (f *fakeRepository) List(_ context.Context) ([]Conversation, error) { + return f.convs, nil +} +// ... +``` + +No database needed for handler unit tests. + +-------------------------------------------------------------------------------- + +## 7. File layout + +``` +sidecar/ +├── main.go # Pool init, migrate, wire plugins +├── BUILD # bazel build file +│ +├── db/ +│ ├── querier.go # Querier interface + WithTx helper +│ ├── migrate/ +│ │ ├── migrate.go # Migration runner +│ │ ├── 0001_create_conversations.sql +│ │ └── ... +│ └── BUILD +│ +└── todo/ + ├── repository.go # Repository interface + pgxRepository + ├── server.go # HTTP handlers (HandleList, HandleCreate) + ├── types.go # Conversation domain struct + ├── repository_test.go # pgxmock-based unit tests + ├── server_test.go # httptest-based handler tests + └── BUILD +``` + +Future plugins follow the same layout: `sidecar//`. + +-------------------------------------------------------------------------------- + +## 8. Open questions + +| Question | Recommendation | +| ------------------------- | ------------------------------------------ | +| Which PostgreSQL server? | Local `postgres` process managed by the | +: : gateway CLI, or a long-running system : +: : postgres. The CLI should `pg_isready` : +: : check it on startup and fail fast if : +: : absent. : +| Connection string source | CLI flag `--pg-dsn` or per-workspace | +: : config in : +: : `~/.agent_manager/workspaces/.json`. : +: : Flag is simpler for now. : +| pgx major version | v5 (current). It changes the `Rows.Scan` | +: : ergonomics vs. v4 — use `pgx/v5/pgxpool`. : +| Transaction wrapping in | Handlers that need atomicity call | +: handlers : `db.WithTx(ctx, pool, func(q db.Querier) : +: : error { ... })`. Repositories are : +: : constructed inside the callback with : +: : `todo.NewRepository(q)`. : +| `pgxmock` availability in | Check the module registry for `pgxmock` or | +: the project : `pgx/v5/pgxmock`. If absent, use a : +: : hand-written fake implementing : +: : `db.Querier`. : + +-------------------------------------------------------------------------------- + +## 9. Summary: what is actually being built + +| Component | Necessary? | Rationale | +| ------------------------------ | ---------- | ----------------------------- | +| `db.Querier` interface | ✅ Yes | Testability seam; keeps pool | +: : : concrete, hidden from plugins : +| `db.WithTx` helper | ✅ Yes | Reusable transaction | +: : : boilerplate : +| `db/migrate` package | ✅ Yes | Schema evolution without an | +: : : external tool : +| `todo.Repository` interface | ✅ Yes | Decouples handler logic from | +: : : SQL; enables fakes in tests : +| `todo.pgxRepository` | ✅ Yes | The one concrete | +: : : implementation doing the SQL : +: : : work : +| Generic "DB wrapper" struct | ❌ No | Pointless indirection; | +: : : pgxpool.Pool is already good : +| Generic ORM / reflection-based | ❌ No | Hand-written scan functions | +: mapper : : are simpler and type-safe : diff --git a/dataclass_array/memory/design/2026-03-27-sidecar-database-abstraction-v4.md b/dataclass_array/memory/design/2026-03-27-sidecar-database-abstraction-v4.md new file mode 100644 index 0000000..91dfbed --- /dev/null +++ b/dataclass_array/memory/design/2026-03-27-sidecar-database-abstraction-v4.md @@ -0,0 +1,559 @@ +# Sidecar Database Abstraction + +> **Status**: Design draft — 2026-03-27 (rev 4) **Previous**: +> `2026-03-26-sidecar-database-abstraction-v3.md` + +-------------------------------------------------------------------------------- + +## 1. Problem Statement + +The v2 sidecar returns stub empty arrays. We need a real persistence layer. +Constraints: + +- **All workspaces share one PostgreSQL database.** Each workspace has its own + sidecar process, but all connect to the same Postgres. Concurrency is a + first-class concern. +- **`Conversation` is a sidecar core concept.** Any plugin can annotate a + conversation with plugin-specific metadata. No plugin owns it. +- **Plugins are hardcoded** — there is no runtime plugin discovery. The plugin + interface exists only where polymorphism pays for itself. + +-------------------------------------------------------------------------------- + +## 2. Plugin Registration: Hardcoded, Not Dynamic + +### The question + +The v3 design had a full `Plugin` interface: `ID()`, `SchemaSQL()`, +`RegisterRoutes()`, `GetConversationMeta()`. The user's question was: **does +this pay for itself, or would hardcoding be cleaner?** + +### Decision: intermediate approach + +Three options were considered: + +| Option | How it works | Verdict | +| ------------------ | --------------------------------- | ------------------ | +| **A — Full dynamic | All plugin concerns behind one | Over-abstracted. | +: interface** : interface; registered via a slice : Registry loop for : +: : : schema/routes adds : +: : : no value when : +: : : plugins are : +: : : hardcoded : +| **B — Fully | `main.go` calls | Simple but leaves | +: hardcoded** : `todomanager.ApplySchema(...)`, : no structure for : +: : `todomanager.RegisterRoutes(...)` : the one case where : +: : directly : polymorphism is : +: : : real : +| **C — Minimal | Schema + routes wired explicitly | Best of both | +: interface, : in `main.go`; interface used only : : +: explicit wiring** : for conversation meta assembly : : + +**We use Option C.** The interface is scoped to the one operation that is +genuinely polymorphic: assembling plugin annotations when serving a +conversation. Everything else (schema, routes) is explicit: + +```go +// sidecar/main.go + +func main() { + pool, _ := db.Open(ctx, *dbDSN) + + // ── Schema — explicit, each plugin owns its own idempotent SQL ────── + db.ApplyCoreSchema(ctx, pool) + todomanager.ApplySchema(ctx, pool) + // future plugins added here + + mux := http.NewServeMux() + mux.HandleFunc("/api/health", handleHealth) + + // ── Routes — explicit ──────────────────────────────────────────────── + todoPlugin := todomanager.New(pool) + todoPlugin.RegisterRoutes(mux) + // future plugins added here + + // ── Conversation handler — polymorphic plugin list ─────────────────── + // Only the conversation handler needs to iterate plugins at runtime. + plugins := []plugin.ConversationPlugin{todoPlugin} + mux.Handle("/api/conversations", newConversationHandler(pool, plugins)) + mux.Handle("/api/conversations/", newConversationHandler(pool, plugins)) + + http.ListenAndServe(fmt.Sprintf(":%d", *port), mux) +} +``` + +### The minimal interface + +```go +// sidecar/plugin/plugin.go +package plugin + +import "context" + +// ConversationPlugin is the only interface plugins must implement. +// It exists solely to support the conversation metadata assembly loop +// in GET /api/conversations and GET /api/conversations/:id. +// +// Schema application and HTTP route registration are handled explicitly +// in main.go — they do not belong to an interface. +type ConversationPlugin interface { + // PluginID returns the plugin's unique key (e.g. "todo-manager"). + // Used as the key in the response's "plugins" map. + PluginID() string + + // BatchGetConversationMeta returns plugin metadata for the given + // conversation IDs. Missing IDs are omitted from the map. + // Called once per LIST request, once per GET request. + BatchGetConversationMeta(ctx context.Context, ids []string) (map[string]any, error) +} +``` + +This is the entire interface. `SchemaSQL() string` is gone — it was only needed +by the registry loop, which is now explicit in `main.go`. + +-------------------------------------------------------------------------------- + +## 3. Core: `Conversation` + +### 3.1 Single struct + +The v3 design had two types: `Conversation` and `ConversationWithPlugins`. The +user asked why. **There is no good reason** — collapse them into one: + +```go +// sidecar/store/conversation.go +package store + +import "time" + +// Conversation is the sidecar's core record for any Antigravity conversation. +// Plugin-specific data is stored in plugin extension tables and optionally +// included here in the Plugins map when serving API responses. +type Conversation struct { + ID string `db:"id" json:"id"` // Antigravity cascade ID + WorkspaceID string `db:"workspace_id" json:"workspaceId"` + CreatedAt time.Time `db:"created_at" json:"createdAt"` + Plugins map[string]any `db:"-" json:"plugins,omitempty"` // assembled at query time +} +``` + +`Plugins` is `nil` when the struct is used internally (e.g. in the repo before +annotation). It is always populated before serialising to JSON. + +### 3.2 Core schema + +```sql +-- sidecar/db/core_schema.sql + +CREATE TABLE IF NOT EXISTS conversations ( + id TEXT PRIMARY KEY, -- Antigravity cascade ID (/c/) + workspace_id TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE INDEX IF NOT EXISTS conversations_workspace_idx ON conversations(workspace_id); +``` + +### 3.3 Repository + +```go +// sidecar/store/conversation_repo.go + +type ConversationRepo struct{ pool *pgxpool.Pool } + +// Create inserts a new conversation. Returns ErrConflict if the ID already +// exists — a duplicate cascade ID is a bug, not a routine race. +func (r *ConversationRepo) Create(ctx context.Context, id, workspaceID string) (Conversation, error) { + var c Conversation + err := r.pool.QueryRow(ctx, + `INSERT INTO conversations (id, workspace_id) + VALUES ($1, $2) + RETURNING id, workspace_id, created_at`, + id, workspaceID, + ).Scan(&c.ID, &c.WorkspaceID, &c.CreatedAt) + if isUniqueViolation(err) { + return Conversation{}, fmt.Errorf("conversation %s already exists: %w", id, ErrConflict) + } + return c, err +} + +func (r *ConversationRepo) ListAll(ctx context.Context) ([]Conversation, error) { ... } +func (r *ConversationRepo) Get(ctx context.Context, id string) (Conversation, error) { ... } +``` + +Duplicate cascade IDs returning `ErrConflict` (HTTP 409) surfaces bugs instead +of swallowing them. + +### 3.4 Handler — assembling the plugin map + +Plugin metadata is fetched in **batch per plugin** (not per conversation) to +avoid N+1: + +```go +func (h *conversationHandler) assemblePlugins(ctx context.Context, convos []Conversation) error { + ids := convIDs(convos) + byID := make(map[string]*Conversation, len(convos)) + for i := range convos { byID[convos[i].ID] = &convos[i] } + + for _, p := range h.plugins { + metas, err := p.BatchGetConversationMeta(ctx, ids) + if err != nil { + return fmt.Errorf("plugin %s: %w", p.PluginID(), err) + } + for convID, meta := range metas { + c := byID[convID] + if c.Plugins == nil { c.Plugins = map[string]any{} } + c.Plugins[p.PluginID()] = meta + } + } + return nil +} +``` + +-------------------------------------------------------------------------------- + +## 4. Plugin-Specific Data: Extension Tables + +Each plugin's schema SQL lives in the **plugin's own folder** and is applied +explicitly from `main.go`. No interface method needed. + +### 4.1 `todo_attempt` table (renamed from `todo_manager_conversation_meta`) + +The table is renamed `todo_attempt` because it captures the concept of a +**single attempt at completing a todo** — it is not just metadata. + +A `todo_attempt` also gets its own `attempt_status` to record the lifecycle of +that specific attempt, independent of Antigravity's agent state stream (which is +ephemeral). + +```sql +-- sidecar/todo_manager/schema.sql + +-- ── Todo status ────────────────────────────────────────────────────────── +-- +-- Stored as TEXT, validated in Go. No DB ENUM or CHECK constraint yet. +-- Rationale: status values are still evolving. A CHECK constraint is easy +-- to add later once values stabilize; an ENUM is harder to ALTER. +-- See §4.2 for the tradeoff discussion. + +CREATE TABLE IF NOT EXISTS todo_manager_todos ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + title TEXT NOT NULL, + parent_id UUID REFERENCES todo_manager_todos(id) ON DELETE CASCADE, + status TEXT NOT NULL DEFAULT 'not_started', + content TEXT NOT NULL DEFAULT '', + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- ── todo_attempt ───────────────────────────────────────────────────────── +-- +-- Each row = one attempt to complete a todo via a Antigravity conversation. +-- A todo can have many attempts (multiple tries). A conversation currently +-- maps to at most one attempt, but this is enforced by a UNIQUE constraint +-- that can be dropped later if many-to-many is needed. + +CREATE TABLE IF NOT EXISTS todo_attempt ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + todo_id UUID NOT NULL REFERENCES todo_manager_todos(id) ON DELETE CASCADE, + conversation_id TEXT NOT NULL REFERENCES conversations(id) ON DELETE CASCADE, + attempt_status TEXT NOT NULL DEFAULT 'pending', + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + + UNIQUE (conversation_id) -- one conversation → at most one attempt (relaxable later) +); + +CREATE INDEX IF NOT EXISTS todo_attempt_todo_idx ON todo_attempt(todo_id); +``` + +### 4.2 `attempt_status` values + +Value | Meaning +----------- | ------------------------------------------- +`pending` | Conversation created, agent not yet started +`running` | Agent actively working +`succeeded` | Agent completed the task +`failed` | Agent errored or was stopped mid-task +`abandoned` | User manually dismissed this attempt + +### 4.3 `todo_status` vs `attempt_status` — ENUM or TEXT? + +**The tradeoff:** + +| Approach | Adding a value | Removing a value | Type safety | +| --------------- | ------------------ | ---------------- | ----------------- | +| PostgreSQL | `ALTER TYPE ADD | Impossible | DB-enforced | +: `ENUM` : VALUE` — : without : : +: : non-transactional, : recreating : : +: : can't roll back : : : +| Lookup table | `INSERT INTO | `DELETE FROM | FK-enforced | +: : statuses VALUES : statuses VALUES : : +: : (...)` : (...)` : : +| `TEXT` + CHECK | `ALTER TABLE DROP | Same | DB-enforced | +: constraint : CONSTRAINT + ADD : : : +: : CONSTRAINT` : : : +| `TEXT` + Go | Edit Go code | Edit Go code | App-enforced only | +: validation only : : : : + +**Decision for now: `TEXT` validated in Go only.** + +The status values for both `todo_status` and `attempt_status` are still evolving +rapidly. During this phase, enforcing validity in the application layer (a Go +constant set) is sufficient and avoids painful DDL churn. + +**Planned upgrade path:** once the values stabilize, add a PostgreSQL CHECK +constraint. If performance or data integrity requirements grow, move to a lookup +table. ENUM is not recommended because removing values is impractical. + +### 4.4 Go types + +```go +// sidecar/todo_manager/types.go +package todomanager + +import "time" + +// TodoStatus enumerates valid todo status values. +// Validated in Go; stored as TEXT in the database. +type TodoStatus string + +const ( + TodoStatusNotStarted TodoStatus = "not_started" + TodoStatusInProgress TodoStatus = "in_progress" + TodoStatusNeedsReview TodoStatus = "needs_review" + TodoStatusDone TodoStatus = "done" +) + +// AttemptStatus enumerates valid attempt status values. +type AttemptStatus string + +const ( + AttemptStatusPending AttemptStatus = "pending" + AttemptStatusRunning AttemptStatus = "running" + AttemptStatusSucceeded AttemptStatus = "succeeded" + AttemptStatusFailed AttemptStatus = "failed" + AttemptStatusAbandoned AttemptStatus = "abandoned" +) + +type Todo struct { + ID string `db:"id"` + Title string `db:"title"` + ParentID *string `db:"parent_id"` + Status TodoStatus `db:"status"` + Content string `db:"content"` + CreatedAt time.Time `db:"created_at"` + UpdatedAt time.Time `db:"updated_at"` + Children []Todo `db:"-"` // assembled in-process from flat list +} + +// TodoAttempt is a single attempt at completing a todo. +type TodoAttempt struct { + ID string `db:"id"` + TodoID string `db:"todo_id"` + ConversationID string `db:"conversation_id"` + Status AttemptStatus `db:"attempt_status"` + CreatedAt time.Time `db:"created_at"` + UpdatedAt time.Time `db:"updated_at"` + Todo *Todo `db:"-"` // optionally populated +} + +// AttemptMeta is the plugin's contribution to the conversation's plugins map. +// Appears as: {"todo-manager": AttemptMeta}. +type AttemptMeta struct { + AttemptID string `json:"attemptId"` + TodoID string `json:"todoId"` + AttemptStatus AttemptStatus `json:"attemptStatus"` + Todo *Todo `json:"todo,omitempty"` +} +``` + +### 4.5 Plugin struct (implements `ConversationPlugin`) + +```go +// sidecar/todo_manager/plugin.go +package todomanager + +type Plugin struct { repo *Repo } + +func New(pool *pgxpool.Pool) *Plugin { return &Plugin{repo: NewRepo(pool)} } + +func (p *Plugin) PluginID() string { return "todo-manager" } + +func (p *Plugin) RegisterRoutes(mux *http.ServeMux) { + h := newHandler(p.repo) + mux.HandleFunc("/api/todo-manager/todos", h.handleTodos) + mux.HandleFunc("/api/todo-manager/todos/", h.handleTodo) + mux.HandleFunc("/api/todo-manager/attempts", h.handleAttempts) + mux.HandleFunc("/api/todo-manager/attempts/", h.handleAttempt) +} + +func (p *Plugin) BatchGetConversationMeta(ctx context.Context, ids []string) (map[string]any, error) { + attempts, err := p.repo.BatchGetAttemptsByConversation(ctx, ids) + if err != nil { + return nil, err + } + result := make(map[string]any, len(attempts)) + for convID, attempt := range attempts { + result[convID] = AttemptMeta{ + AttemptID: attempt.ID, + TodoID: attempt.TodoID, + AttemptStatus: attempt.Status, + Todo: attempt.Todo, + } + } + return result, nil +} +``` + +-------------------------------------------------------------------------------- + +## 5. Concurrency: Multiple Sidecars, One Database + +| Situation | Mechanism | +| ----------------------------------- | -------------------------------------- | +| Schema DDL at startup (N sidecars | `pg_advisory_lock` — one executes DDL, | +: racing) : others wait, then no-op on `IF NOT : +: : EXISTS` : +| Inserting a new conversation | Plain `INSERT` — fail with | +: : `ErrConflict` (409) on duplicate : +: : cascade ID : +| Creating a todo | Plain `INSERT` — no conflict possible | +: : across sidecars : +| Updating a todo (optimistic) | `UPDATE ... WHERE id=$1 AND | +: : updated_at=$2` — 0 rows → 409 : +| Updating attempt_status | `UPDATE todo_attempt SET | +: : attempt_status=$1, updated_at=now() : +: : WHERE id=$2` — last writer wins : +: : (status updates are monotonic) : +| Read-then-mutate (reparent subtree) | `SELECT ... FOR UPDATE` + explicit | +: : transaction : + +-------------------------------------------------------------------------------- + +## 6. API + +### 6.1 Core conversation endpoints + +``` +POST /api/conversations body: {id, workspaceId} + → 201 {id, workspaceId, createdAt, plugins: {}} + → 409 if id already exists + +GET /api/conversations → [{id, workspaceId, createdAt, plugins: {...}}] +GET /api/conversations/:id → {id, workspaceId, createdAt, plugins: {...}} +``` + +### 6.2 Todo manager endpoints + +``` +GET /api/todo-manager/todos → todo tree (children assembled in-process) +POST /api/todo-manager/todos → create {title, parentId?} +PUT /api/todo-manager/todos/:id → update {title?, status?, content?} +DELETE /api/todo-manager/todos/:id → delete (cascades to attempts) + +POST /api/todo-manager/attempts → create {todoId, conversationId} +GET /api/todo-manager/attempts?todoId= → list attempts for a todo +PUT /api/todo-manager/attempts/:id → update {attemptStatus} +``` + +The `POST /api/todo-manager/attempts` handler wraps both the conversation insert +(if the conversation doesn't exist yet) and the `todo_attempt` insert in one +transaction. + +### 6.3 Response shape + +```ts +interface Conversation { + id: string; // Antigravity cascade ID + workspaceId: string; + createdAt: string; + plugins: { + "todo-manager"?: { + attemptId: string; + todoId: string; + attemptStatus: "pending" | "running" | "succeeded" | "failed" | "abandoned"; + todo?: { id: string; title: string; status: string; ... }; + }; + }; +} +``` + +-------------------------------------------------------------------------------- + +## 7. Postgres Lifecycle + +**The Gateway owns Postgres** — same supervision pattern as sidecar and LS. + +``` +Gateway startup: + 1. Start/attach Postgres + 2. Wait for connection + 3. Start sidecar (receives --db_dsn flag, opens pool, applies schema) + 4. Start LS +``` + +-------------------------------------------------------------------------------- + +## 8. File Layout + +``` +sidecar/ + main.go # Explicit wiring: schema, routes, plugin list + db/ + db.go # Open(ctx, dsn) *pgxpool.Pool + schema.go # ApplyCoreSchema (advisory lock + embedded SQL) + core_schema.sql # conversations table only + store/ + conversation.go # Conversation type (with Plugins field) + conversation_repo.go # ConversationRepo + errors.go # ErrNotFound, ErrConflict + plugin/ + plugin.go # ConversationPlugin interface only + todo_manager/ + schema.sql # todo_manager_todos + todo_attempt + schema.go # ApplySchema(ctx, pool) — called from main + types.go # Todo, TodoAttempt, AttemptMeta, status consts + repo.go # TodoRepo + AttemptRepo + handler.go # HTTP handlers + plugin.go # Implements plugin.ConversationPlugin + BUILD +``` + +-------------------------------------------------------------------------------- + +## 9. What We Are NOT Building + +| Omitted | Why | +| -------------------------------- | ---------------------------------------- | +| Dynamic plugin discovery | Plugins are hardcoded — the interface is | +: : scoped to conversation meta only : +| `SchemaSQL() string` on Plugin | Schema applied explicitly; no need for | +: interface : interface polymorphism here : +| `ConversationWithPlugins` second | Merged into `Conversation.Plugins` | +: struct : : +| `ON CONFLICT DO NOTHING` on | Conflict is a bug → explicit 409 error | +: conversations : : +| PostgreSQL ENUM for status | Too brittle during rapid iteration; TEXT | +: : + Go constants for now : +| `metadata JSONB` blob | Typed extension tables are queryable and | +: : safe : +| Per-workspace database | One DB, concurrency via PostgreSQL | +| Migration library | Idempotent `IF NOT EXISTS` sufficient | +: : while schema is young : + +-------------------------------------------------------------------------------- + +## 10. Still Open + +| Question | Notes | +| ----------------------------------- | -------------------------------------- | +| How does the gateway route absolute | Gateway design doc TBD — options: | +: `/api/` requests to the correct : Referer header, injected `X-Workspace` : +: workspace's sidecar? : header, session cookie : +| When does `attempt_status` get | Sidecar polls agent state? Frontend | +: updated? : PATCHes it? Needs decision : +| Pagination on `GET | Add `?limit=&offset=` when list grows | +: /api/conversations` : : +| CHECK constraint threshold | Add once both status enums have been | +: : stable for ≥2 weeks : diff --git a/dataclass_array/memory/design/2026-03-28-gateway-v2-abstractions.md b/dataclass_array/memory/design/2026-03-28-gateway-v2-abstractions.md new file mode 100644 index 0000000..b1b449d --- /dev/null +++ b/dataclass_array/memory/design/2026-03-28-gateway-v2-abstractions.md @@ -0,0 +1,305 @@ +# Gateway v2 — Full Abstraction Redesign + +> Companion to [gateway-v2-redesign.md](2026-03-28-gateway-v2-redesign.md), +> which covers the `WorkspaceInfo`/`WorkspaceStore` data model. +> +> This document covers how **every other gateway abstraction** changes to work +> with that new model. + +--- + +## Abstractions at a glance + +``` +┌─────────────────────────────────────────────────────────────┐ +│ main.go — wiring + flag parsing │ +│ │ +│ Creates: │ +│ WorkspaceStore ──owns──▶ head.json + workspaces/*.json │ +│ Supervisor ──uses──▶ WorkspaceStore (read-only) │ +│ Router ──uses──▶ WorkspaceStore (read-only) │ +│ ──uses──▶ Supervisor (start services) │ +│ │ +│ Supervisor ──creates──▶ workspaceProxy (per workspace) │ +└─────────────────────────────────────────────────────────────┘ +``` + +**Dependency direction:** `WorkspaceStore` is the bottom of the stack. It knows +nothing about Supervisor, Router, or Proxy. Everything reads from it; nothing +writes to it except `main.go` (on startup) and the `POST /api/gateway/head` +handler. + +--- + +## 1. `WorkspaceStore` + +> Replaces: `Registry` + `HeadPointer` + +Owns all workspace state: the per-workspace JSON files and the HEAD pointer. +Self-contained — no knowledge of processes, ports, or routing. + +```go +type WorkspaceStore struct { /* unexported fields */ } + +// Construction +func NewWorkspaceStore(dir string, headPath string, cacheTTL time.Duration) (*WorkspaceStore, error) + +// --- Raw reads (from disk / cache) --- + +func (s *WorkspaceStore) Get(name string) *WorkspaceEntry // nil if not registered +func (s *WorkspaceStore) List() []*WorkspaceEntry // all registered (raw) +func (s *WorkspaceStore) Head() *WorkspaceEntry // current HEAD entry (always complete), nil if unset + +// --- Resolved reads (merged with HEAD fallback) --- + +func (s *WorkspaceStore) Resolve(name string) (*WorkspaceInfo, error) + // Returns workspace.Info.Merge(head.Info). + // Errors if workspace not registered, or if HEAD is unset. + +// --- Writes --- + +func (s *WorkspaceStore) SetHead(name string) error + // 1. Reads workspace entry for `name` (must exist). + // 2. Merges: newHead = workspace.Info.Merge(oldHead.Info) + // → workspace paths win; missing fields keep old HEAD values. + // 3. Writes head.json atomically. + // 4. Errors if the resulting HEAD is incomplete (any path nil). +``` + +### Key invariant + +**HEAD is always complete.** Every read of `Head()` returns a `WorkspaceEntry` +where all three paths are non-nil. This is enforced: + +- On startup: if `head.json` does not exist, the gateway **must** be started + with `--set-workspace-head=` pointing to a workspace that has all 3 + paths. If any path is missing → fatal error. +- On `SetHead()`: the merge result must have all 3 paths. If not → error + returned (HEAD not updated). + +This eliminates nil-checking downstream. Any caller of `Resolve()` knows it +will get a fully populated `WorkspaceInfo`. + +--- + +## 2. `Supervisor` + +> Replaces: current `Supervisor` (which hard-codes blaze-bin path logic) + +Owns process lifecycle: spawn, restart, reap. Does **not** own binary path +resolution — it receives fully resolved paths from callers. + +```go +type Supervisor struct { /* unexported fields */ } + +// Construction & lifecycle +func (s *Supervisor) Start(ctx context.Context) error +func (s *Supervisor) Stop() +func (s *Supervisor) StartReaper(ctx context.Context, idleTimeout time.Duration) + +// --- Service management --- + +func (s *Supervisor) GetOrStartSidecar(name string, binPath string) (port int, err error) + // Starts (or returns existing) sidecar for workspace `name`. + // `binPath` is the resolved sidecar binary path. + +func (s *Supervisor) GetOrStartLanguageServer(name string, binPath string, bundlePath string) (port int, err error) + // Starts (or returns existing) LS for workspace `name`. + // `binPath` is the resolved LS binary. + // `bundlePath` is the resolved frontend dist/ path. + +func (s *Supervisor) RestartService(workspaceName, serviceName string) error + +// --- Observability --- + +func (s *Supervisor) TouchInstance(name string) +func (s *Supervisor) GetStatus(workspaces []*WorkspaceEntry) []WorkspaceStatus + // Takes a list of workspaces instead of a *Registry. + +// --- Proxy cache --- + +func (s *Supervisor) GetProxy(name string) *workspaceProxy +func (s *Supervisor) SetProxy(name string, proxy *workspaceProxy) +``` + +### What changed + +| Before | After | +|--------|-------| +| `Supervisor.SidecarBin` / `LSBin` fields | Deleted. No default paths stored on supervisor. | +| `UpdateDefaultsFromWorkspace()` | Deleted. Callers pass resolved paths directly. | +| `GetOrStartSidecar(workspace *WorkspaceEntry)` — discovers blaze-bin paths internally | `GetOrStartSidecar(name, binPath)` — receives resolved path. | +| `GetOrStartLanguageServer(workspace, bundlePath)` — discovers blaze-bin paths internally | `GetOrStartLanguageServer(name, binPath, bundlePath)` — receives all paths. | +| `GetStatus(reg *Registry)` — reaches into Registry | `GetStatus(workspaces)` — receives data, no dependency on store. | + +The supervisor is now a **pure process manager**. It has zero knowledge of +workspace JSON files, HEAD, or binary path conventions. + +--- + +## 3. `workspaceProxy` + +> Unchanged in structure, but constructed differently. + +```go +type workspaceProxy struct { /* unchanged */ } + +func newWorkspaceProxy(sidecarPort, lsPort int, workspace string) *workspaceProxy +func (p *workspaceProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) +``` + +No changes needed. The proxy is already a pure routing object (sidecar port, +LS port, workspace name). The only difference is that the **caller** that +creates it now gets ports from a supervisor that received resolved paths. + +--- + +## 4. Router (`server.go`) + +> Replaces: current `registerRoutes` with its inline process-starting logic. + +The router's job: map URL → workspace → start services if needed → proxy. + +```go +func registerRoutes(mux *http.ServeMux, store *WorkspaceStore, sup *Supervisor) +``` + +### Route table (unchanged semantics) + +| Pattern | Handler | +|---------|---------| +| `GET /api/gateway/health` | Gateway health check | +| `GET /api/gateway/workspaces` | List registered workspaces | +| `GET /api/gateway/status` | Process status for all workspaces | +| `POST /api/gateway/restart` | Restart a service in a workspace | +| `GET /api/gateway/head` | Return current HEAD workspace name | +| `POST /api/gateway/head` | Set HEAD | +| `GET //*` | Proxy to workspace services | +| `GET /head/*` | Alias for HEAD workspace | +| `GET /` | Absolute-path leak detector | + +### Key flow: resolving a workspace request + +``` +GET /feat-x/api/todos + +1. segment = "feat-x" +2. entry = store.Get("feat-x") → existence check (is it registered?) +3. info = store.Resolve("feat-x") → merged WorkspaceInfo (fills from HEAD) +4. scPort = sup.GetOrStartSidecar("feat-x", *info.SidecarPath) +5. lsPort = sup.GetOrStartLanguageServer("feat-x", *info.LSPath, *info.FrontendPath) +6. proxy = newWorkspaceProxy(scPort, lsPort, "feat-x") +7. proxy.ServeHTTP(w, r) +``` + +Steps 3–6 only run on first request (proxy is cached via `sup.GetProxy`). + +### `POST /api/gateway/head` handler + +``` +1. Validate workspace exists: store.Get(name) != nil +2. Update HEAD: store.SetHead(name) + → Merges workspace.Info with old HEAD, writes head.json. + → Errors if result incomplete. +3. Return {"workspace": name} +``` + +Note: no `sup.UpdateDefaultsFromWorkspace()` call. The supervisor doesn't track +defaults anymore. Next workspace request will call `store.Resolve()` which reads +the updated HEAD. + +--- + +## 5. `main.go` — wiring + +```go +func main() { + flag.Parse() + + store := NewWorkspaceStore(registryDir, headPath, 2*time.Second) + + // HEAD bootstrap: required on first run. + if *setWorkspaceHead != "" { + store.SetHead(*setWorkspaceHead) // fatals if incomplete + } + if store.Head() == nil { + log.Fatal("No HEAD workspace set. Use --set-workspace-head=") + } + + sup := &Supervisor{} + sup.Start(ctx) + go sup.StartReaper(ctx, 10*time.Minute) + + mux := http.NewServeMux() + registerRoutes(mux, store, sup) + + // ... serve +} +``` + +### What changed + +- `Registry` and `HeadPointer` are replaced by a single `WorkspaceStore`. +- `sup.UpdateDefaultsFromWorkspace(headEntry)` is deleted. +- HEAD is required — fatal if missing after flag processing. + +--- + +## 6. `buildLSCmd` — stays in `main.go` + +```go +func buildLSCmd(lsBin string, lsPort int, bundlePath string, workspaceName string) []string +``` + +Unchanged. It's a pure function that assembles LS flags. No dependency changes. + +--- + +## Dependency graph (after) + +```mermaid +graph TD + Main["main.go"] --> Store["WorkspaceStore"] + Main --> Sup["Supervisor"] + Main --> Router["registerRoutes()"] + Router --> Store + Router --> Sup + Router --> Proxy["workspaceProxy"] + Sup --> ManagedProc["managedProc"] + Store --> WI["WorkspaceInfo / Merge()"] +``` + +**Before**, `Supervisor` depended on `Registry` (via `GetStatus`) and stored +`SidecarBin`/`LSBin` state. Now it is a leaf — it depends on nothing except +its own `managedProc` type. + +--- + +## File mapping: old → new + +| Old file | New file | Notes | +|----------|----------|-------| +| `registry.go` | `store.go` | Merged with head.go into WorkspaceStore | +| `head.go` | `store.go` | HEAD is now part of WorkspaceStore | +| `supervisor.go` | `supervisor.go` | Simplified: receives resolved paths | +| `server.go` | `server.go` | Uses WorkspaceStore instead of Registry+HeadPointer | +| `proxy.go` | `proxy.go` | Unchanged | +| `main.go` | `main.go` | Simplified wiring | +| *(new)* | `workspace.go` | `WorkspaceInfo`, `WorkspaceEntry`, `Merge()` | +| `registry_test.go` | `store_test.go` | | +| `head_test.go` | `store_test.go` | Merged | +| `main_test.go` | `main_test.go` | Unchanged | +| *(new)* | `workspace_test.go` | `Merge()` tests | + +--- + +## Summary of what each abstraction knows + +| Abstraction | Knows about | Does NOT know about | +|---|---|---| +| `WorkspaceInfo` | Three nullable paths | Files, processes, ports, routing | +| `WorkspaceEntry` | Name + WorkspaceInfo | Files, processes, ports, routing | +| `WorkspaceStore` | Dir layout, JSON files, HEAD file, merge semantics | Processes, ports, proxy, routing | +| `Supervisor` | Process lifecycle, ports, restart logic | Workspace files, HEAD, path resolution | +| `workspaceProxy` | Sidecar port, LS port, routing `/api/*` vs `/*` | Everything else | +| `Router` | URL → workspace mapping, composing store + supervisor | Binary paths (delegates to store) | diff --git a/dataclass_array/memory/design/2026-03-28-gateway-v2-redesign.md b/dataclass_array/memory/design/2026-03-28-gateway-v2-redesign.md new file mode 100644 index 0000000..c85332e --- /dev/null +++ b/dataclass_array/memory/design/2026-03-28-gateway-v2-redesign.md @@ -0,0 +1,254 @@ +# Gateway WorkspaceEntry Redesign — Abstraction Design + +## Problem statement + +The Go gateway still uses the old `WorkspaceEntry` shape (`path`, `bundle_path`,) while the Python CLI was updated to the new three-path model +(`ls_path`, `sidecar_path`, `frontend_path`). The two are out of sync. + +Beyond the schema mismatch, the current design scatters merge-and-fallback +logic across `Registry`, `HeadPointer`, and `Supervisor`. This redesign +extracts that into one self-contained abstraction. + +--- + +## New data model + +### `WorkspaceInfo` — the shared, nullable three-path record + +```go +// WorkspaceInfo is the JSON-serialisable shape shared with the Python CLI. +// Every field is optional: only the paths that have been built are present. +type WorkspaceInfo struct { + LSPath *string `json:"ls_path,omitempty"` + SidecarPath *string `json:"sidecar_path,omitempty"` + FrontendPath *string `json:"frontend_path,omitempty"` +} +``` + +- Mirrors the Python `WorkspaceEntry` field-for-field (minus `name`, which is + the filename key). +- `nil` pointer = "not built for this workspace yet". +- Serialises with `omitempty` so absent fields are truly absent in JSON. + +### `WorkspaceEntry` — identity + info + +```go +type WorkspaceEntry struct { + Name string // derived from filename, never serialised + Info WorkspaceInfo // raw (non-merged) data from this workspace's JSON +} +``` + +The `Name` field is NOT written to the JSON file (it is the filename stem), +matching Python's `to_dict()` which already omits it. + +--- + +## Merge semantics + +``` +Merged(workspace, head) = workspace.Info, with nil fields filled from head.Info +``` + +```go +// Merge fills nil fields in base with values from fallback. +// Returns a new WorkspaceInfo; neither input is mutated. +func (base WorkspaceInfo) Merge(fallback WorkspaceInfo) WorkspaceInfo { + out := base + if out.LSPath == nil { + out.LSPath = fallback.LSPath + } + if out.SidecarPath == nil { + out.SidecarPath = fallback.SidecarPath + } + if out.FrontendPath == nil { + out.FrontendPath = fallback.FrontendPath + } + return out +} +``` + +This is pure — no I/O, no side effects, easy to test. + +--- + +## `WorkspaceStore` — self-contained workspace management abstraction + +The `WorkspaceStore` owns everything that touches the workspace JSON files. +It replaces the current `Registry` + `HeadPointer.Resolve()` responsibilities. + +```go +type WorkspaceStore struct { + dir string // ~/.agent_manager/workspaces/ + headPath string // ~/.agent_manager/head.json (not "head", see below) + + mu sync.RWMutex + cache map[string]*WorkspaceEntry + cacheTime time.Time + cacheTTL time.Duration +} +``` + +### Key methods + +```go +// List returns all registered workspace entries (raw, non-merged). +func (s *WorkspaceStore) List() []*WorkspaceEntry + +// Get returns a workspace entry by name (raw, non-merged), or nil. +func (s *WorkspaceStore) Get(name string) *WorkspaceEntry + +// Resolve returns the effective WorkspaceInfo for a workspace, +// with nil fields filled from HEAD. +// Returns nil only if the workspace does not exist. +func (s *WorkspaceStore) Resolve(name string) *WorkspaceEntry + +// Head returns the current HEAD WorkspaceEntry (raw), or nil. +func (s *WorkspaceStore) Head() *WorkspaceEntry + +// ResolvedHead returns the HEAD entry with nil fields filled from... itself. +// (HEAD is always complete, so this is mostly a passthrough.) +func (s *WorkspaceStore) ResolvedHead() *WorkspaceEntry + +// SetHead atomically writes name as the head and updates head.json. +func (s *WorkspaceStore) SetHead(name string) error +``` + +### `head.json` — always complete + +HEAD is stored as a JSON file (`~/.agent_manager/head.json`) rather than a +plain text pointer file. It contains: + +1. `name` — which workspace is HEAD. +2. A complete `WorkspaceInfo` — the **merged** info at the time it was last + updated (all three paths present). + +```json +{ + "name": "refactor-head-to-gateway", + "ls_path": "/path/to/language_server", + "sidecar_path": "/path/to/sidecar", + "frontend_path": "/path/to/dist" +} +``` + +This means: +- HEAD is **always** the fallback source of truth. +- If a workspace only built its sidecar, the gateway fills in LS and frontend + from HEAD automatically. + +### Updating HEAD + +When a workspace becomes HEAD (`SetHead`): + +``` +head.Info = head.Info.Merge(workspace.Info) +``` + +i.e. the workspace's paths *override* the old HEAD values, but only for the +paths that are present. The remaining paths stay as they were in HEAD. + +This keeps HEAD always complete (assuming it was complete when first populated). + +--- + +## How Supervisor changes + +The current `Supervisor` hard-codes binary path discovery logic (blaze-bin +paths). That logic moves to `WorkspaceStore.Resolve()` — the supervisor just +receives a resolved `WorkspaceInfo` and reads the paths directly. + +```go +func (s *Supervisor) GetOrStartSidecar(workspace *WorkspaceEntry) (int, error) { + info := store.Resolve(workspace.Name) + if info.SidecarPath == nil { + return 0, fmt.Errorf("sidecar not built for %s", workspace.Name) + } + return s.getOrStart(workspace.Name, "sidecar", *info.SidecarPath, ...) +} +``` + +The `Supervisor.UpdateDefaultsFromWorkspace()` method is deleted — it is +replaced by the `Resolve()` call above, which always fetches the correct +fallback from HEAD. + +--- + +## What changes in the gateway `server.go` + +The `server.go` routing code now calls `store.Resolve(name)` instead of +`reg.Get(name)` when it needs the effective binary paths. The routing itself +(workspace prefix → which workspace) still uses `store.Get()` for existence +checks. + +--- + +## File layout (proposed) + +``` +gateway/ + workspace.go # WorkspaceInfo, WorkspaceEntry, Merge() + store.go # WorkspaceStore (replaces registry.go + head.go) + supervisor.go # updated to use WorkspaceStore + server.go # updated to use WorkspaceStore + proxy.go # unchanged + main.go # updated wiring +``` + +--- + +## Issues I spotted that you may have overlooked + +### 1. `head.json` vs plain `head` text file +The current gateway stores HEAD as a plain text file (workspace name only). +The new design proposes `head.json` to store the complete merged info alongside +the name. **Migration needed**: on startup, if `head.json` is missing but +`head` (plain text) exists, the gateway should migrate it. + +### 2. Head bootstrap problem +If `head.json` does not exist yet (first run), HEAD paths are all nil. The +system cannot start. You need either: + - A `--head=` flag on `./gateway` that populates HEAD from a + registered workspace. + - Or the Python CLI's `start` command explicitly calls `SetHead` before + launching the gateway. + +Currently `--set-workspace-head` does this, but only if the workspace is +already registered. That flow will need to be preserved. + +### 3. Frontend path vs bundle path +The current gateway uses `BundlePath` (passed to the LS command) and `Path` +(the worktree root). In the new model, `FrontendPath` replaces both — it is +the compiled `dist/` directory, which is also the bundle the LS serves. +Double-check that the LS `--jetbox_bundle_path` flag expects the `dist/` +directory (not the worktree root). + +### 4. The `frontend_path` typo in the user story +The user's description uses `frontent_path` (missing 'd') — Python registry +uses `frontend_path`. This design uses `frontend_path` to match Python. + +### 5. Reaper / idle timeout still needs `WorkspaceEntry.Name` +The reaper accesses instances by name. This is unaffected by the redesign +(`WorkspaceEntry.Name` remains the key). + +### 6. `handleListWorkspaces` response shape +The current endpoint returns the raw `WorkspaceEntry` array. After the +redesign, do you want the API to return raw or resolved entries? Recommendation: +return raw per-workspace data plus HEAD info separately so the frontend can +show what each workspace has built. + +--- + +## Verification plan + +### Unit tests +- `workspace_test.go`: pure `Merge()` tests — nil propagation, all combos. +- `store_test.go`: replaces `registry_test.go` + `head_test.go`. + - Write workspace JSON with partial fields, read back, assert merge. + - SetHead → assert head.json written correctly. + +### Integration +- Build sidecar only → workspace entry has only `sidecar_path`. +- Gateway resolves it: sidecar comes from workspace, LS + frontend from HEAD. +- Build frontend → workspace entry gains `frontend_path`. +- Gateway now serves workspace frontend, falls back sidecar/LS from HEAD. diff --git a/dataclass_array/memory/design/2026-03-30-plugin-testing-design.md b/dataclass_array/memory/design/2026-03-30-plugin-testing-design.md new file mode 100644 index 0000000..e604496 --- /dev/null +++ b/dataclass_array/memory/design/2026-03-30-plugin-testing-design.md @@ -0,0 +1,431 @@ +# Plugin Testing Design + +**Date:** 2026-03-30 **Author:** epot + +-------------------------------------------------------------------------------- + +## 1. Problem statement + +The plugin system lives in **XX** (`plugins/` + `plugins-core/`) but the +test infrastructure lives in the **YY** (`exa/agent_ui_toolkit/`). Our +plugins are compiled *into* the GoB bundle via Vite aliases (`@plugins → +.../plugins/`, `@plugins-core → .../plugins-core/`), so at build time they are +just ordinary TypeScript modules inside the GoB project. + +This split creates three testing risks identified during the grey-screen +incident: + +| Risk | Trigger mechanism | What failed | +| --------------------- | ------------------------ | ---------------------- | +| Missing Redux context | Plugin `effect` uses | `ConversationRecorder` | +: : `useSelector` : crashed in isolation : +| Wrong injection point | `` placed | Structural regression, | +: : outside provider : grey screen : +| Silent no-op wiring | `wirePlugins` called | Sidebar contributions | +: : before featureManager : swallowed : +: : init : : + +TypeScript doesn't catch any of these — they are *runtime invariant* violations. +We need actual component tests. + +-------------------------------------------------------------------------------- + +## 2. Available test infrastructure (GoB repo) + +The GoB repo already has a fully configured, working Vitest + React Testing +Library setup with **jsdom** and no network access: + +``` +exa/agent_ui_toolkit/ +├── vitest.config.ts # jsdom env, globals:true, @testing-library/react +├── vitest.setup.ts # jest-dom matchers, ResizeObserver mock, matchMedia mock +├── node_modules/ +│ ├── vitest@4 # ✅ test runner +│ ├── @testing-library/react@16 # ✅ render / screen / fireEvent +│ ├── @testing-library/jest-dom # ✅ custom matchers +│ ├── @reduxjs/toolkit # ✅ configureStore +│ └── react-redux # ✅ Provider +``` + +**`npm run test`** (`vitest run`) already picks up every file matching +`src/**/*.test.{ts,tsx}` | `dev/**/*.test.{ts,tsx}`. + +The Vite *build* and the Vitest *test* runner both resolve `@plugins` and +`@plugins-core` via aliases, but **the vitest config currently has neither +alias**. This is the only gap to bridge. + +-------------------------------------------------------------------------------- + +## 3. The alias gap — and how to close it + +### 3a. Why the gap exists + +`vitest.config.ts` only configures: + +```ts +include: ['src/**/*.test.{ts,tsx}', 'dev/**/*.test.{ts,tsx}', ...] +// No resolve.alias for @plugins or @plugins-core +``` + +Our plugin tests import from `@plugins/...` or `../../plugins/...` (relative). +The relative path can work if tests sit inside the worktree where plugins are +symlinked (see §4), but `@plugins` aliases fail without explicit configuration. + +### 3b. Proposed approach — patch `vitest.config.ts` + +We already have a patch system (`patches/`). The cleanest approach is a new +patch file: **`patches/plugins_vitest.patch`**. + +The patch adds `resolve.alias` entries pointing to the same absolute source +paths that `vite.config.ts` uses: + +```diff +--- exa/agent_ui_toolkit/vitest.config.ts ++++ exa/agent_ui_toolkit/vitest.config.ts ++import path from 'path'; ++ + export default defineConfig({ + test: { + globals: true, + environment: 'jsdom', + setupFiles: ['./vitest.setup.ts'], + include: [ + 'src/**/*.test.{ts,tsx}', + 'dev/**/*.test.{ts,tsx}', ++ '../../..//agent_manager/v2/plugins/**/*.test.{ts,tsx}', ++ '../../..//agent_manager/v2/plugins-core/**/*.test.{ts,tsx}', + ], + }, ++ resolve: { ++ alias: { ++ '@plugins': '__PLUGINS_DIR__', ++ '@plugins-core': '__PLUGINS_CORE_DIR__', ++ '@': path.resolve(__dirname, 'src'), ++ }, ++ }, + }); +``` + +`__PLUGINS_DIR__` and `__PLUGINS_CORE_DIR__` are already template placeholders +substituted by `builder.py` when applying patches (same as in +`plugins_vite.patch.tpl`). + +This means: - Tests run from **inside the worktree** (where `npm run test` +already works). - Plugin test files live in **the source tree** and are included via the +`include` glob. - No new toolchain, no new npm packages. + +### 3c. Alternative — keep tests closer to source + +Instead of reaching into the source tree from the vitest include path, test files can +live in the **worktree itself** (created by the build step). The builder would +copy / symlink a `tests/` directory from `v2/plugins/` into the worktree. This +is more flexible but adds builder complexity. Recommended only if the absolute +path glob proves fragile. + +-------------------------------------------------------------------------------- + +## 4. Test file structure and location + +### Convention + +Test files live **next to the source files they test** in the source tree: + +``` +v2/ +├── plugins-core/ +│ ├── PluginEffects.tsx +│ ├── PluginEffects.test.tsx ← NEW +│ ├── wire.ts +│ └── wire.test.ts ← NEW +└── plugins/ + └── conversations/ + ├── plugin.tsx + └── plugin.test.tsx ← NEW +``` + +This is idiomatic for Vitest (similar to Jest's `*.test.tsx` convention used +throughout the GoB repo) and keeps tests co-located with what they verify. + +### Naming + +- `.test.tsx` for React component tests. +- `.test.ts` for pure logic tests. + +-------------------------------------------------------------------------------- + +## 5. Test categories and exact test cases + +### 5a. Component isolation tests (`plugins-core/PluginEffects.test.tsx`) + +**Goal:** Verify `PluginEffects` can be rendered in isolation (no context +required) and that it mounts effects from all plugins. + +```tsx +import {render} from '@testing-library/react'; +import {describe, it, expect, vi} from 'vitest'; +import {PluginEffects} from './PluginEffects'; +import type {Plugin} from './types'; + +describe('PluginEffects', () => { + it('renders without crashing with an empty plugin list', () => { + // PluginEffects itself needs no context — it just renders children. + render(); + }); + + it('mounts an effect component for every plugin that declares one', () => { + const mounted: string[] = []; + const EffectA = () => { mounted.push('a'); return null; }; + const EffectB = () => { mounted.push('b'); return null; }; + const plugins: Plugin[] = [ + {id: 'a', effect: EffectA}, + {id: 'b', effect: EffectB}, + {id: 'c'}, // No effect — should be skipped. + ]; + render(); + expect(mounted).toEqual(['a', 'b']); + }); +}); +``` + +**Why this matters:** If `PluginEffects` accidentally acquired a Redux +dependency in the future, this test would fail in isolation and surface the +regression immediately. + +### 5b. Plugin context requirement tests (`plugins/conversations/plugin.test.tsx`) + +**Goal:** Document and enforce that `ConversationRecorder` *requires* a Redux +`Provider`. Written as a "crash test" ─ the component must throw (or +console.error) without a store, and must not throw with one. + +```tsx +import {render} from '@testing-library/react'; +import {describe, it, expect, vi, beforeEach, afterEach} from 'vitest'; +import {Provider} from 'react-redux'; +import {configureStore} from '@reduxjs/toolkit'; +import {conversationSlice} from '@/common/...'; // or a minimal stub +import {conversationsPlugin} from './plugin'; + +describe('ConversationRecorder (conversations plugin effect)', () => { + let consoleError: ReturnType; + + beforeEach(() => { + consoleError = vi.spyOn(console, 'error').mockImplementation(() => {}); + }); + afterEach(() => { consoleError.mockRestore(); }); + + it('throws / errors without a Redux Provider', () => { + // useSelector outside a Provider throws in React 18. + expect(() => { + render(); + }).toThrow(); // or check consoleError was called + }); + + it('renders successfully inside a Redux Provider', () => { + const store = configureStore({ + reducer: { + conversation: (state = {convoCreationMessages: []}) => state, + }, + }); + // Should not throw. + render( + + + , + ); + }); +}); +``` + +This test serves as **living documentation**: any future plugin that uses +`useSelector` must be wrapped in the same `Provider` pattern, and the test makes +that requirement explicit. + +### 5c. Structural placement test (optional integration test) + +**Goal:** Catch "PluginEffects placed outside the Provider" regressions. This is +harder to test at the unit level because it requires rendering the actual GoB +`WebEffects.tsx` component tree. Two options: + +**Option A — Smoke test in `dev/` folder (inside GoB):** Write a test in +`dev/WebEffects.test.tsx` inside the worktree. This test lives in a `.patch` +file: `patches/plugins_web_effects_test.patch`. The patch adds the test file and +the include glob. The downside is that the patch adds a test whose source isn't +in the source tree, creating a drift risk. + +**Option B — Contract test on `PluginEffects` return type:** Statically verify +that `WebEffects` always returns `` by scanning its import graph. +This is a simpler approach that doesn't require a new patch, but it only catches +a missing import, not a wrong insertion point. + +**Recommendation:** Start with Option A for maximum protection, but only add +that patch once the baseline (§5a and §5b) is proven working. + +### 5d. Wire logic tests (`plugins-core/wire.test.ts`) + +**Goal:** Verify `wirePlugins` pushes contributions into the featureManager +correctly, and is idempotent-safe. + +```ts +import {describe, it, expect, beforeEach} from 'vitest'; +import {wirePlugins} from './wire'; +import {PluginRegistry} from './registry'; +import type {Plugin, PluginFeatureManager} from './wire'; + +describe('wirePlugins', () => { + let fm: PluginFeatureManager; + + beforeEach(() => { + // Reset PluginRegistry state between tests (it's a module-level singleton). + PluginRegistry.auxSideBarPanes.length = 0; + PluginRegistry.sidebarSections.length = 0; + PluginRegistry.conversationActions.length = 0; + PluginRegistry.topLevelBarItems.length = 0; + fm = {customAuxPanes: new Map(), sidebarItemsFeature: {items: []}}; + }); + + it('wires a sidebar plugin into sidebarItemsFeature', () => { + const SidebarComp = () => null; + const plugin: Plugin = { + id: 'test', + sidebar: {label: 'Test', icon: 'star', component: SidebarComp}, + }; + wirePlugins(fm, [plugin]); + expect(fm.sidebarItemsFeature!.items).toHaveLength(1); + expect(fm.sidebarItemsFeature!.items[0].id).toBe('test'); + }); + + it('wires an auxPane into featureManager.customAuxPanes', () => { + const descriptor = {} as any; + const plugin: Plugin = {id: 'p', auxPane: {id: 'myPane', descriptor}}; + wirePlugins(fm, [plugin]); + expect(fm.customAuxPanes!.get('myPane')).toBe(descriptor); + }); + + it('is a no-op for plugins with no contributions', () => { + wirePlugins(fm, [{id: 'empty'}]); + expect(fm.sidebarItemsFeature!.items).toHaveLength(0); + }); +}); +``` + +-------------------------------------------------------------------------------- + +## 6. Mocking strategy for Redux-dependent plugins + +Plugins using `useSelector` need a Redux store. The recommended pattern is a +**minimal stub store** (not the real Antigravity store) created with +`configureStore`: + +```tsx +// test-utils/makeStore.tsx (lives in v2/plugins-core/ or v2/plugins/) +import {configureStore} from '@reduxjs/toolkit'; + +/** Builds a minimal Redux store for plugin tests. */ +export function makeTestStore(preloadedState: Record = {}) { + return configureStore({ + reducer: Object.fromEntries( + Object.entries(preloadedState).map(([key, val]) => [ + key, + (state = val) => state, + ]), + ), + preloadedState, + }); +} + +/** Renders children inside a test Redux Provider. */ +export function renderWithStore( + ui: React.ReactElement, + preloadedState: Record = {}, +) { + const store = makeTestStore(preloadedState); + return render({ui}); +} +``` + +This avoids importing the real Antigravity Redux state (which has dozens of slices +and circular deps) while still satisfying `react-redux`'s context requirement. + +-------------------------------------------------------------------------------- + +## 7. Running tests + +### During development (inside a built worktree) + +```bash +# Navigate to the active worktree (created by `./agent_manager.py build --build-frontend`) +cd ~/.agent_manager/worktrees//exa/agent_ui_toolkit + +# Run all tests including plugin tests +npm run test + +# Watch mode for TDD +npm run test:watch -- --reporter=verbose +``` + +### Via the CLI (proposed `./agent_manager.py test` integration) + +The `agent_manager.py test` command (already mentioned in `README.md`) should: +1. Resolve the current worktree path. 2. `cd` into `exa/agent_ui_toolkit/`. 3. +Run `npm run test`. + +This is a single-liner addition to `cli/builder.py`. + +-------------------------------------------------------------------------------- + +## 8. Patch plan summary + +| Patch file | What it does | +| ---------------------------------------- | --------------------------------- | +| `patches/plugins_vitest.patch.tpl` | Adds `@plugins` + `@plugins-core` | +: : aliases to `vitest.config.ts` and : +: : extends `include` to source tree : +: : plugin dirs : +| *(no new patch)* | Test files live in source tree | +: : next to source — included via : +: : `include` glob : +| `patches/plugins_web_effects_test.patch` | *(Optional phase 2)* Adds | +: : structural smoke test for : +: : `WebEffects.tsx` : + +-------------------------------------------------------------------------------- + +## 9. Open questions + +1. **Absolute source path in `include` glob**: Vitest's `include` supports + absolute globs, but this creates a coupling between the worktree path and + the source tree path. The builder already knows both — `__PLUGINS_DIR__` and + `__PLUGINS_CORE_DIR__` are safe substitution targets. However, the glob must + also work at runtime inside the worktree. **Validation needed:** try the + glob in an existing worktree before writing the patch. + +2. **PluginRegistry singleton reset**: `PluginRegistry` is a module-level + singleton (`const PluginRegistry = { ... }`). Tests that call `wirePlugins` + will accumulate state across tests unless explicitly reset. Need to either + expose a `reset()` function or restructure tests to be order-independent. + +3. **`conversationSlice` import**: The `ConversationRecorder` test needs to + import or stub the real Redux slice that provides + `state.conversation.convoCreationMessages`. The real slice likely lives deep + in the GoB repo. Using a stub reducer avoids the import but hides future + selector-path breakage. A dedicated slice mock is safer long-term. + +4. **Error boundary interference**: `PluginEffects` is supposed to wrap errors + in an error boundary. The "crash without Provider" test (§5b) may not throw + at the `render()` call level if the error boundary silently catches the + error. Need to confirm: does React 18 + react-redux throw synchronously or + only log via `console.error`? The test should be written accordingly. + +-------------------------------------------------------------------------------- + +## 10. Recommended implementation order + +1. ✅ **Validate** that `npm run test` works in the current worktree as-is. +2. **Write `wire.test.ts`** (pure logic, no React, no Redux — easiest). +3. **Write `PluginEffects.test.tsx`** (React, no Redux — verifies isolation). +4. **Write `plugin.test.tsx` for `conversations`** (React + Redux stub). +5. **Write `patches/plugins_vitest.patch.tpl`** to extend the include glob and + add aliases. +6. **Rebuild worktree** and run `npm run test` to confirm all tests pass. +7. **(Optional)** Add structural `WebEffects.test.tsx` via a second patch. +8. **Update `agent_manager.py test`** to run `npm run test` in the worktree. diff --git a/dataclass_array/memory/design/2026-04-01-gateway-graceful-restart.md b/dataclass_array/memory/design/2026-04-01-gateway-graceful-restart.md new file mode 100644 index 0000000..5179e5d --- /dev/null +++ b/dataclass_array/memory/design/2026-04-01-gateway-graceful-restart.md @@ -0,0 +1,266 @@ +# Gateway Graceful Restart (Pragmatic Approach) + +**Status:** Draft +**Date:** 2026-04-01 + +## Problem + +When the gateway is restarted (e.g. after `./agent_manager.py restart --build-gateway`), it +kills the Language Server and Sidecar processes it supervises. Any agent session running _inside_ +that gateway — including the one that triggered the restart — is interrupted. The browser must wait +several seconds for all child processes to start again before the UI is usable. + +### Why the gateway needs restarting at all + +The gateway binary is only rebuilt when the Go source under `gateway/` changes. Frontend, Sidecar, +and LS changes are hot-deployed without any restart. Gateway restarts are therefore infrequent, but +when they happen, they are currently disruptive enough that the developer must manually re-open the +browser tab and wait for the stack to come back up. + +## Goals + +- The Language Server and Sidecar processes survive a gateway restart. +- After the new gateway is up, the browser reconnects within ~100 ms (a single TCP reconnect). +- Existing streaming RPC connections (gRPC-web) are reset once at the TCP level but immediately + re-established by the browser; the LS session itself is intact. +- No architectural complexity beyond what a single-developer tool warrants. + +## Non-Goals + +- True zero-downtime (no TCP connection drop). This requires file-descriptor passing between + processes (`SCM_RIGHTS`) and is not worth the complexity for a dev tool. +- Preserving the supervisor's in-memory crash history across restarts. +- Full supervision (auto-restart, crash detection) of adopted child processes. Adopted processes + are unmanaged after re-adoption; they must be explicitly restarted via the UI or CLI if they + crash. + +## Design + +### Invariant: children outlive the gateway + +The key change is that **the gateway does not kill its children on exit**. Today, `sup.Stop()` +sends `SIGINT` to all supervised processes before `os.Exit(0)`. After this change, the gateway +exits without touching its children. The OS re-parents them to PID 1 (init), where they continue +running undisturbed. + +### New abstraction: `RuntimeStore` + +All the state needed to re-adopt an orphaned child is written to disk _at spawn time_, before the +child is started. The data lives in `~/.agent_manager/runtime//` and is fully managed +by a new, self-contained module: `gateway/runtime_store.go`. + +``` +~/.agent_manager/ + namespaces/ ← existing: namespace build paths (NamespaceInfo JSON) + runtime/ ← new: per-namespace live process state + head/ + sidecar.json ← {pid, port, csrf_token} (csrf_token absent for sidecar) + language_server.json + feat-x/ + sidecar.json + language_server.json +``` + +`RuntimeStore` is the only component that reads or writes these files. Its public surface is +minimal: + +```go +// ServiceRecord is the persisted state of one running child process. +// It is written before the child starts and deleted when the child is stopped +// intentionally (i.e. not on gateway exit). +type ServiceRecord struct { + PID int `json:"pid"` + Port int `json:"port"` + CSRFToken string `json:"csrf_token,omitempty"` // only Language Server +} + +type RuntimeStore struct { dir string } + +func NewRuntimeStore(dir string) (*RuntimeStore, error) + +// Write atomically writes a record for (namespace, service). +func (r *RuntimeStore) Write(namespace, service string, rec ServiceRecord) error + +// Read returns the persisted record for (namespace, service), or nil if absent. +func (r *RuntimeStore) Read(namespace, service string) (*ServiceRecord, error) + +// Delete removes the record for (namespace, service). +func (r *RuntimeStore) Delete(namespace, service string) error + +// ReadAll returns all records grouped by namespace name. +func (r *RuntimeStore) ReadAll() (map[string]map[string]ServiceRecord, error) +``` + +`RuntimeStore` has no dependency on the `Supervisor` or `NamespaceStore`. It is a pure persistence +layer for `ServiceRecord` values. + +### Supervisor changes: write before spawn, read on startup + +#### On spawn + +`Supervisor.getOrStartProcWithBinPath` writes a `ServiceRecord` to `RuntimeStore` _before_ +calling `cmd.Start()`. If `cmd.Start()` fails, the record is deleted. + +The CSRF token is generated once per LS spawn and stored in the `ServiceRecord`. The token is read +back from the record during re-adoption so the new gateway can proxy to the unchanged LS correctly. + +`buildLSCmd` is updated to accept the token as a parameter instead of generating it internally. +`generateCSRFToken` remains in `main.go` and is called by the Supervisor before spawning. + +#### On shutdown + +`sup.Stop()` is **removed** from the signal handler. The gateway exits cleanly without sending any +signal to children. `RuntimeStore` records are intentionally _not_ deleted on exit — they are the +handoff to the next gateway instance. + +Records _are_ deleted when the Supervisor intentionally stops a namespace (e.g. +`RestartNamespace`), since in that case the children are killed on purpose. + +#### On startup: `Supervisor.AdoptOrphans` + +A new method is called once, during gateway startup, after the `Supervisor` is initialised: + +```go +// AdoptOrphans reads all ServiceRecords from the RuntimeStore and attempts to +// re-adopt any processes that are still alive (verified via kill(pid, 0)). +// Adopted instances are registered in the supervisor's instances map with +// IsAdopted=true. Stale records (dead PID) are deleted from the RuntimeStore. +func (s *Supervisor) AdoptOrphans(rs *RuntimeStore) error +``` + +For each live record, `AdoptOrphans`: + +1. Creates a `NamespaceInstance` and `serviceInstance` in the supervisor's `instances` map. +2. Sets `svc.Port` and `svc.BinPath` from the record — the port is what matters for proxying. +3. Sets `svc.Proc = nil` (no `exec.Cmd` — we did not spawn the process). +4. Sets `svc.IsAdopted = true` (new field on `serviceInstance`). +5. Does **not** start a `supervise()` goroutine — adopted processes are unmanaged. + +The proxy (`namespaceProxy`) is constructed immediately from the adopted ports, so the first +browser request after startup is served without any cold-start delay. + +### `serviceInstance` gets `IsAdopted bool` + +```go +type serviceInstance struct { + Proc *managedProc + Port int + BinPath string + IsAdopted bool // true when process was inherited from a previous gateway +} +``` + +`IsAdopted` is surfaced in `ServiceStatus` (the JSON struct returned by `/api/gateway/status`): + +```go +type ServiceStatus struct { + Name string `json:"name"` + Port int `json:"port"` + PID int `json:"pid"` + Up bool `json:"up"` + BinPath string `json:"bin_path"` + IsAdopted bool `json:"is_adopted"` // new + Crashes []CrashEvent `json:"crashes"` +} +``` + +The UI can use `is_adopted` to display a visual indicator (e.g. "⟳ adopted" badge next to the +sidecar/LS status) with a "Restart" button that calls the existing +`POST /api/gateway/restart_from_head` endpoint to replace the adopted processes with fresh, +fully-supervised ones. + +### `processUpAndPID` for adopted processes + +`processUpAndPID` currently inspects `svc.Proc.proc.Process`. For adopted processes, `svc.Proc` +is nil. A new helper `adoptedProcessUp(pid int) bool` uses `syscall.Kill(pid, 0)` to check +liveness. `processUpAndPID` is updated to fall back to this when `IsAdopted` is true. + +### Startup sequence (updated) + +``` +main() + ├─ NewNamespaceStore(...) // unchanged + ├─ NewRuntimeStore(...) // new + ├─ sup.Start(ctx) // unchanged + ├─ sup.AdoptOrphans(runtimeStore) // new: re-adopt live children + ├─ sup.StartReaper(ctx, ...) // unchanged + ├─ registerRoutes(mux, store, sup) // unchanged + └─ srv.ListenAndServe() // unchanged +``` + +### Shutdown sequence (updated) + +``` +SIGTERM / Ctrl-C + └─ cancel() // cancel context → supervise() goroutines exit + // children are NOT killed + // RuntimeStore records are NOT deleted + // os.Exit(0) (or srv.Shutdown) +``` + +### Port binding: `SO_REUSEPORT` + +To eliminate the ~100 ms TCP gap during the handover window, the gateway binds port 3001 with +`SO_REUSEPORT`. This allows the new gateway process to start accepting connections before the old +one has fully exited. In practice the gap is zero at the kernel level; the browser experiences no +connection error. + +Implementation: replace `srv.ListenAndServe()` with a manually created `net.Listener` using +`net.ListenConfig` with `Control` set to apply `SO_REUSEPORT` via `syscall.SetsockoptInt`. + +This is a small, self-contained change in `main.go` (~15 lines). + +### `agent_manager.py restart` sequence (updated) + +``` +1. build (optional, --build-gateway etc.) +2. send SIGTERM to old gateway ← gateway exits; children survive +3. wait for port 3001 to be free (with SO_REUSEPORT this may be instant) +4. start new gateway +5. new gateway calls AdoptOrphans → proxy is ready immediately +6. browser reconnects → requests served without cold-start +``` + +Steps 2–4 already exist in `cmd_stop` / `cmd_start`. No Python changes are required. + +## File Map + +| File | Change | +|---|---| +| `gateway/runtime_store.go` | **New.** `RuntimeStore`, `ServiceRecord`. | +| `gateway/supervisor.go` | Add `IsAdopted` to `serviceInstance`; add `AdoptOrphans`; write/delete `ServiceRecord` on spawn/stop; remove `SIGINT` fan-out from `Stop()`. | +| `gateway/main.go` | Add `NewRuntimeStore`; call `AdoptOrphans`; switch to `SO_REUSEPORT` listener; remove `sup.Stop()` from signal handler. | +| `gateway/server.go` | Expose `is_adopted` in `ServiceStatus`. | +| `gateway/namespace.go` | No change. | +| `gateway/store.go` | No change. | +| `gateway/proxy.go` | No change. | + +## Known Limitations + +| Limitation | Impact | Mitigation | +|---|---|---| +| Adopted processes have no auto-restart | If the LS crashes after adoption, it stays down silently | `is_adopted` UI badge + "Restart" button; user calls `restart_from_head` | +| Adopted processes are not killed on new gateway exit | If two consecutive restarts happen before the user clicks "Restart", stale children accumulate | `AdoptOrphans` cleans up dead PIDs; `restart_from_head` kills and respawns | +| Crash history is lost | Observability gap | Accepted; crash history is a debug aid, not critical | +| `SO_REUSEPORT` is Linux-only | N/A for this use case (Cloud Workstations) | None needed | +| Streaming gRPC-web connections reset once | Single reconnect spinner | Accepted; LS session is intact | + +## Alternatives Considered + +### Full zero-downtime (file-descriptor passing) + +Pass the listening socket's file descriptor from old to new gateway via a Unix socket using +`SCM_RIGHTS`. The new gateway inherits the live listener with zero TCP gap. Rejected: ~1 week of +implementation, adds significant complexity to `main.go`, not warranted for a dev tool. + +### Killing children but restarting them faster + +Keep the current kill-on-exit behaviour but make the LS start faster. Not viable: the LS cold +start is ~3–5 s and is controlled by the Antigravity team's binary, not ours. + +### Nginx / Caddy as the outer proxy + +Replacing the Go gateway with a standard reverse proxy would give us graceful reload for free +(`nginx -s reload`). Rejected: the gateway's namespace-routing logic, referer fallback, and +supervisor are not expressible in Nginx config without custom Lua modules. See earlier research +(conversation `e81d3927`). diff --git a/dataclass_array/memory/design/2026-04-01-streaming-rpc-investigation.md b/dataclass_array/memory/design/2026-04-01-streaming-rpc-investigation.md new file mode 100644 index 0000000..bdbdc3f --- /dev/null +++ b/dataclass_array/memory/design/2026-04-01-streaming-rpc-investigation.md @@ -0,0 +1,311 @@ +# Streaming RPC Investigation — Agent Manager v2 + +_Date: 2026-04-01_ + +## Summary + +The frontend communicates with the Language Server (LS) using **gRPC-Web** via +the **ConnectRPC** framework (not raw gRPC over HTTP/2). All RPC calls, +including streaming ones, go through the **Go gateway** reverse proxy on +`:3001`. The gateway forwards them to the LS over plain HTTP/1.1. + +--- + +## Full Call Stack + +``` +Browser (React) + │ gRPC-Web (Connect protocol, HTTP/1.1, JSON format) + │ POST /head/exa.language_server_pb.LanguageServerService/JetboxSubscribeToState + ▼ +Gateway (:3001, plain HTTP/1.1) + │ httputil.ReverseProxy (Go stdlib, HTTP/1.1) + │ Strips /head/ prefix → forwards to LS port + ▼ +Language Server (:300X, plain HTTP/1.1) + │ ConnectRPC handler (Go) + │ JetboxSubscribeToState → server-streaming response + ▼ +Browser receives chunked stream +``` + +--- + +## Frontend Client Setup + +### Library + +| Package | Version | Role | +|--------------------------|---------|-------------------------------------| +| `@connectrpc/connect` | `^2.0.0`| RPC client core | +| `@connectrpc/connect-web`| `^2.0.0`| gRPC-Web / Connect protocol adapter | +| `@bufbuild/protobuf` | – | Proto serialization | + +### Transport Initialization + +File: `exa/agent_ui_toolkit/src/clients/StandaloneAgentClient.ts` + +```typescript +const transport = createGrpcWebTransport({ + baseUrl, // see below + interceptors: [csrfInterceptor], + useBinaryFormat: false, // uses JSON (Content-Type: application/grpc-web+json) +}); +this.lsClient = createClient(LanguageServerService, transport); +``` + +> **`useBinaryFormat: false`** → The wire format is `application/grpc-web+json` +> (not binary protobuf). Each RPC request is a standard HTTP POST. + +### Base URL Computation + +The `baseUrl` is constructed by the `client_config.patch` we inject: + +```typescript +// Original upstream code: +let baseUrl = config.baseUrl || window.location.origin; + +// Our patch appends the namespace prefix: +const namespace = (window as any).__NAMESPACE__; +if (namespace && !config.baseUrl) { + baseUrl = `${baseUrl}/${namespace}`; +} +``` + +The `__NAMESPACE__` global is injected by the gateway's `proxy.go` into every +`index.html` response: + +```go +script := fmt.Sprintf(``, namespace) +``` + +So the actual `baseUrl` seen by ConnectRPC is: + +``` +http://:3001/head +``` + +### RPC URL Pattern + +ConnectRPC constructs request URLs as: + +``` +POST {baseUrl}/{fully-qualified-service}/{method} +``` + +For `JetboxSubscribeToState`: + +``` +POST http://:3001/head/exa.language_server_pb.LanguageServerService/JetboxSubscribeToState +``` + +The gateway strips `/head/` → forwards as: + +``` +POST http://localhost:/exa.language_server_pb.LanguageServerService/JetboxSubscribeToState +``` + +--- + +## Streaming RPC: `JetboxSubscribeToState` + +### Proto definition + +```proto +// In language_server.proto: +rpc JetboxSubscribeToState(JetboxSubscribeToStateRequest) + returns (stream JetboxSubscribeToStateResponse) {} +``` + +This is a **server-streaming RPC**: one request, potentially infinite responses. + +### Frontend call site + +File: [`ls-debug/plugin.tsx`](file:///research-v2-streaming-rpc/agent_manager/v2/plugins/ls-debug/plugin.tsx) + +```typescript +const stream = lsClient.jetboxSubscribeToState({}, { signal: abortCtrl.signal }); +for await (const msg of stream) { + // process first message, then break +} +``` + +The ConnectRPC library implements server-streaming over gRPC-Web using +**chunked HTTP/1.1** (or HTTP/2 if available). Since our gateway uses plain +`httputil.ReverseProxy` over HTTP/1.1, the stream arrives as a series of +length-prefixed frames in the response body. + +### Backend handler + +File: `third_party/Antigravity/language_server/rpcs_jetbox_state.go` + +```go +func (s *Server) JetboxSubscribeToState( + ctx context.Context, + connectReq *connect.Request[...], + stream *connect.ServerStream[...], +) error { + updates := make(chan *jetbox_state_pb.State, 10) + s.jetboxStateStore.Subscribe(ctx, func(state *jetbox_state_pb.State) { + select { + case updates <- state: + case <-ctx.Done(): + } + }) + for { + select { + case <-ctx.Done(): return ctx.Err() + case state := <-updates: + stream.Send(...) + } + } +} +``` + +The RPC blocks indefinitely until the context is cancelled (client disconnect) +or an error occurs. **The first message is only sent when +`jetboxStateStore.WriteState()` is called** — the stream hangs until there is +a state write. + +--- + +## Language Server HTTP Server Configuration + +File: `third_party/Antigravity/language_server/server.go` + +The LS runs **two** HTTP servers: + +| Server | Protocol | Port | Notes | +|--------|-------------|----------|------------------------------------| +| HTTPS | HTTP/2 + TLS| separate | For desktop IDE (h2 + ALPN) | +| HTTP | HTTP/1.1 | `--http_server_port` (`:300X`) | For browser / gateway proxy | + +Both servers set `SetUnencryptedHTTP2(true)` in `http.Protocols`, so they +*can* do h2c (cleartext HTTP/2). However, the gateway's +`httputil.ReverseProxy` connects over **plain HTTP/1.1** only (no h2c +negotiation), so in practice the gateway↔LS leg runs HTTP/1.1. + +The LS uses the `connect-go` library (`connectrpc.com/connect`), which natively supports all three protocols: +- **Connect** (POST + JSON or binary, Content-Type: `application/connect+…`) +- **gRPC-Web** (Content-Type: `application/grpc-web+…`) +- **gRPC** (HTTP/2 only, Content-Type: `application/grpc`) + +Since the browser transport uses `createGrpcWebTransport` with +`useBinaryFormat: false`, the wire Content-Type is: + +``` +Content-Type: application/grpc-web+json +``` + +--- + +## Gateway Proxy Behavior for Streaming + +File: [`proxy.go`](file:///research-v2-streaming-rpc/agent_manager/v2/gateway/proxy.go) + +```go +proxy := httputil.NewSingleHostReverseProxy(target) +// ModifyResponse only fires for non-stream, html responses +proxy.ModifyResponse = func(resp *http.Response) error { + if !strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html") { + return nil // gRPC-web responses pass through unmodified + } + // ... inject tag and __NAMESPACE__ script +} +``` + +### Critical issue with streaming + +`httputil.ReverseProxy` uses `http.ResponseWriter.Write()` to forward the +response body. For streaming gRPC-Web, the LS sends a chunked HTTP/1.1 body. +Go's `httputil.ReverseProxy`: + +1. **Does not buffer** the response body — it copies directly. +2. **Does not flush** proactively — it relies on the underlying + `http.ResponseWriter` to detect that the client is still connected. +3. Uses `io.Copy` internally, which reads in 32 KB chunks. For gRPC-Web frames + that are much smaller, this can cause **buffering delays**. + +The `ModifyResponse` hook **reads the entire body** into memory for HTML +responses. For non-HTML responses (gRPC-Web), it returns `nil` immediately, so +streaming is pass-through. However, **if the `ModifyResponse` function is +called on a streaming response with a body that never ends** (e.g., a +connection classified as HTML by mistake), it would hang waiting for +`io.ReadAll`. + +### The `logResponse` middleware wrapper + +```go +srv := &http.Server{ + Addr: fmt.Sprintf(":%d", *port), + Handler: logResponse(mux), // wraps with a loggingResponseWriter +} +``` + +`logResponse` wraps the `ResponseWriter` in a `loggingResponseWriter` that +intercepts `WriteHeader`. This wrapper does NOT interfere with streaming body +writes. + +--- + +## CSRF Token Flow + +The LS is started with `--csrf_token ` (generated per run in +`main.go:generateCSRFToken()`). The frontend reads this token from the injected +` +``` + +This allows: + +1. **Relative Asset Loading**: All relative paths (e.g., + `