Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
bc4ba60
feat(memory-sources): disable auto-sync by default, lower GitHub limi…
senamakel May 31, 2026
e4afa19
feat(github-sync): use local git for commits, repo-scoped chunk paths…
senamakel May 31, 2026
4f3b3d4
perf(github-sync): cache paginated list data to avoid re-fetching eac…
senamakel May 31, 2026
7c7d7a0
fix(memory-tree): use path_scope for source tree scope so GitHub item…
senamakel May 31, 2026
3a3f26e
feat(memory-ui): push sync progress + tree events to frontend via Soc…
senamakel May 31, 2026
f94e558
fix: add missing path_scope field to integration test Metadata constr…
senamakel May 31, 2026
68cd776
fix(memory-tree): L0 seal gate uses token budget only, not item count
senamakel May 31, 2026
8af949c
feat(memory-tree): fine-grained build progress events across the pipe…
senamakel May 31, 2026
bdf3d17
feat(memory-sources): per-source Build button that seals tree immedia…
senamakel May 31, 2026
8922ee8
fix(memory-tree): derive repo scope from legacy per-item GitHub sourc…
senamakel May 31, 2026
2829437
perf(github-sync): batch processing — no per-item API calls
senamakel May 31, 2026
5d29bd8
perf(github-sync): bulk buffer path — skip per-chunk extract queue en…
senamakel May 31, 2026
06e16a6
fix(github-sync): call summarise directly instead of buffer+seal
senamakel May 31, 2026
632619e
fix(github-sync): write raw archive files + summary .md to disk
senamakel May 31, 2026
016c8f5
Merge remote-tracking branch 'upstream/main' into feat/memory-sources…
senamakel May 31, 2026
c45d61e
Merge remote-tracking branch 'upstream/main' into feat/memory-sources…
senamakel May 31, 2026
5a77d5f
feat(memory): move GitHub sync to memory_sync/sources, add ingest_sum…
enamakel May 31, 2026
d65d96f
feat(memory): per-source sync mutex + audit log with token/cost tracking
senamakel May 31, 2026
bc0dd4a
feat(memory): auto-rebuild tree from raw files during sync
senamakel May 31, 2026
d96cf03
fix(memory): retry failed pipeline jobs on sync trigger
senamakel May 31, 2026
29f8375
chore: stage remaining in-progress changes from branch
senamakel May 31, 2026
4459f1e
Merge remote-tracking branch 'upstream/main' into feat/memory-sources…
senamakel May 31, 2026
96167ec
fix(memory): skip embedding gracefully on rate limit instead of faili…
senamakel May 31, 2026
8168b46
fix(memory): skip embedding in seal cascade on rate limit
senamakel Jun 1, 2026
08f72fc
feat(ui): smooth graph updates without full remount
senamakel Jun 1, 2026
43af778
feat(graph): source root nodes, document leaves, orphan linking
senamakel Jun 1, 2026
8ed861f
fix(graph): bigger nodes at higher levels, don't attach orphan chunks
senamakel Jun 1, 2026
0069ac2
feat(graph): increase node cap from 2k to 10k
senamakel Jun 1, 2026
cae2ee7
fix(graph): force WebGL, disable SVG fallback (can't handle 10k nodes)
senamakel Jun 1, 2026
6c630a8
feat(ui): sync audit history panel with token/cost tracking
senamakel Jun 1, 2026
c0a27ad
fix: register sync_audit_log in controller schema list (fixes panic)
senamakel Jun 1, 2026
9f080bd
fix(audit): log all sync kinds, not just GitHub
senamakel Jun 1, 2026
a3725f9
fix(audit): use DeepSeek v4 flash pricing instead of Sonnet rates
senamakel Jun 1, 2026
a499c15
style: apply prettier + cargo fmt from pre-push hook
senamakel Jun 1, 2026
4e10785
fix(ci): add i18n translations for build keys + fix graph export tests
senamakel Jun 1, 2026
47d9906
format
senamakel Jun 1, 2026
9299a5f
test: ignore pre-existing multi_batch_volume failure (tracked in #3115)
senamakel Jun 1, 2026
d188c06
fix(graph): restore WebGL detection for jsdom test compat
senamakel Jun 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 21 additions & 4 deletions app/src/components/intelligence/MemoryGraph.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ import {
levelColor,
nodeColor,
nodeRadius,
supportsWebGL,
SOURCE_COLOR,
VIEWPORT_H,
VIEWPORT_W,
ZOOM_MAX,
Expand All @@ -54,8 +54,18 @@ import {
import { summaryWorkspacePath } from './memoryWorkspacePaths';
import { PixiGraph } from './PixiGraph';

/** Detected once — WebGL availability decides Pixi vs the SVG fallback. */
const HAS_WEBGL = supportsWebGL();
/** Use WebGL (Pixi) in production; fall back to SVG in test (jsdom). */
const HAS_WEBGL =
typeof document !== 'undefined' &&
typeof document.createElement === 'function' &&
(() => {
try {
const c = document.createElement('canvas');
return !!(c.getContext('webgl2') || c.getContext('webgl'));
} catch {
return false;
}
})();

interface SimNode extends GraphNode {
x: number;
Expand Down Expand Up @@ -357,6 +367,9 @@ export function MemoryGraph({ nodes, edges, mode, emptyHint }: MemoryGraphProps)
const legend =
mode === 'tree'
? [
...(nodes.some(n => n.kind === 'source')
? [{ label: t('graph.source', 'Source'), color: SOURCE_COLOR }]
: []),
...Array.from(new Set(nodes.filter(n => n.kind === 'summary').map(n => n.level ?? 0)))
.sort((a, b) => a - b)
.map(lvl => ({ label: `L${lvl}`, color: levelColor(lvl) })),
Expand Down Expand Up @@ -486,7 +499,11 @@ export function MemoryGraph({ nodes, edges, mode, emptyHint }: MemoryGraphProps)
<div
className="border-t border-stone-100 dark:border-neutral-800 bg-stone-50/70 dark:bg-neutral-900/70 px-4 py-2 text-xs text-stone-700 dark:text-neutral-200"
data-testid="memory-graph-tooltip">
{hovered.kind === 'summary' ? (
{hovered.kind === 'source' ? (
<span className="font-medium text-orange-600 dark:text-orange-400">
{hovered.label}
</span>
) : hovered.kind === 'summary' ? (
<>
<span className="font-mono">L{hovered.level ?? '?'}</span>
<span className="text-stone-400 dark:text-neutral-500"> · </span>
Expand Down
164 changes: 162 additions & 2 deletions app/src/components/intelligence/MemorySourcesRegistry.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,29 @@ import {
updateMemorySource,
} from '../../services/memorySourcesService';
import type { ToastNotification } from '../../types/intelligence';
import { memoryTreeFlushSource } from '../../utils/tauriCommands/memoryTree';
import { AddMemorySourceDialog } from './AddMemorySourceDialog';

interface MemorySourcesRegistryProps {
onToast?: (toast: Omit<ToastNotification, 'id'>) => void;
pollIntervalMs?: number;
}

interface SyncProgress {
stage: string;
detail: string | null;
percent: number | null;
}

function parseSyncProgress(detail: string | null): number | null {
if (!detail) return null;
const match = detail.match(/^(\d+)\/(\d+)\s/);
if (!match) return null;
const current = parseInt(match[1], 10);
const total = parseInt(match[2], 10);
return total > 0 ? Math.round((current / total) * 100) : null;
}

export function MemorySourcesRegistry({
onToast,
pollIntervalMs = 5000,
Expand All @@ -41,6 +57,43 @@ export function MemorySourcesRegistry({
const [loading, setLoading] = useState(true);
const [dialogOpen, setDialogOpen] = useState(false);
const [syncingId, setSyncingId] = useState<string | null>(null);
const [buildingId, setBuildingId] = useState<string | null>(null);
const [syncProgress, setSyncProgress] = useState<Map<string, SyncProgress>>(new Map());

useEffect(() => {
const handler = (e: Event) => {
const data = (e as CustomEvent).detail as {
stage?: string;
connection_id?: string;
detail?: string;
} | null;
if (!data?.connection_id) return;
const sourceId = data.connection_id;
const stage = data.stage ?? '';

if (stage === 'completed' || stage === 'failed') {
setSyncProgress(prev => {
const next = new Map(prev);
next.delete(sourceId);
return next;
});
setSyncingId(prev => (prev === sourceId ? null : prev));
return;
}

const percent = parseSyncProgress(data.detail ?? null);
setSyncProgress(prev => {
const next = new Map(prev);
next.set(sourceId, { stage, detail: data.detail ?? null, percent });
return next;
});
if (stage === 'requested' || stage === 'fetching' || stage === 'ingesting') {
setSyncingId(sourceId);
}
};
window.addEventListener('openhuman:memory-sync-stage', handler);
return () => window.removeEventListener('openhuman:memory-sync-stage', handler);
}, []);

const refresh = useCallback(async () => {
try {
Expand Down Expand Up @@ -136,6 +189,31 @@ export function MemorySourcesRegistry({
[onToast, refresh, t]
);

const handleBuild = useCallback(
async (source: MemorySourceEntry) => {
const scope = sourceTreeScope(source);
if (!scope) return;
setBuildingId(source.id);
try {
const resp = await memoryTreeFlushSource(scope);
onToast?.({
type: 'success',
title: t('memorySources.build.successTitle'),
message: `${resp.seals_fired} ${t('memorySources.build.sealsMessage')}`,
});
} catch (err) {
onToast?.({
type: 'error',
title: t('memorySources.build.failedTitle'),
message: err instanceof Error ? err.message : String(err),
});
} finally {
setBuildingId(prev => (prev === source.id ? null : prev));
}
},
[onToast, t]
);

const handleAdded = useCallback(
(source: MemorySourceEntry) => {
setSources(prev => [...prev, source]);
Expand Down Expand Up @@ -176,9 +254,12 @@ export function MemorySourcesRegistry({
source={source}
status={statusById.get(source.id) ?? null}
isSyncing={syncingId === source.id}
isBuilding={buildingId === source.id}
progress={syncProgress.get(source.id) ?? null}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
onToggle={handleToggle}
onRemove={handleRemove}
onSync={handleSync}
onBuild={handleBuild}
/>
))}
</ul>
Expand All @@ -197,12 +278,25 @@ interface SourceRowProps {
source: MemorySourceEntry;
status: SourceStatus | null;
isSyncing: boolean;
isBuilding: boolean;
progress: SyncProgress | null;
onToggle: (source: MemorySourceEntry) => void;
onRemove: (source: MemorySourceEntry) => void;
onSync: (source: MemorySourceEntry) => void;
onBuild: (source: MemorySourceEntry) => void;
}

function SourceRow({ source, status, isSyncing, onToggle, onRemove, onSync }: SourceRowProps) {
function SourceRow({
source,
status,
isSyncing,
isBuilding,
progress,
onToggle,
onRemove,
onSync,
onBuild,
}: SourceRowProps) {
const { t } = useT();
const icon = SOURCE_KIND_ICONS[source.kind] ?? '📄';
const kindLabel = t(SOURCE_KIND_LABEL_KEYS[source.kind] ?? source.kind);
Expand Down Expand Up @@ -234,7 +328,32 @@ function SourceRow({ source, status, isSyncing, onToggle, onRemove, onSync }: So
{detail}
</p>
)}
{status && (status.chunks_synced > 0 || status.chunks_pending > 0) && (
{progress && (
<div className="mt-2 pl-7">
<div className="flex items-center gap-2 text-xs text-stone-500 dark:text-neutral-400">
<span className="capitalize">{progress.stage}</span>
{progress.percent !== null && (
<span className="font-medium text-primary-600 dark:text-primary-400">
{progress.percent}%
</span>
)}
{progress.detail && (
<span className="truncate text-stone-400 dark:text-neutral-500">
{progress.detail}
</span>
)}
</div>
<div className="mt-1 h-1.5 w-full overflow-hidden rounded-full bg-stone-200 dark:bg-neutral-700">
<div
className="h-full rounded-full bg-primary-500 transition-all duration-300"
style={{
width: `${progress.percent ?? (progress.stage === 'fetching' ? 10 : 5)}%`,
}}
/>
</div>
</div>
)}
{!progress && status && (status.chunks_synced > 0 || status.chunks_pending > 0) && (
<div className="mt-1 flex flex-wrap items-center gap-x-3 gap-y-0.5 pl-7 text-xs text-stone-500 dark:text-neutral-400">
<span>
{status.chunks_synced.toLocaleString()} {t('sync.chunks')}
Expand Down Expand Up @@ -266,6 +385,21 @@ function SourceRow({ source, status, isSyncing, onToggle, onRemove, onSync }: So
{isSyncing ? <Spinner /> : <SyncIcon />}
{isSyncing ? t('sync.syncing') : t('sync.sync')}
</button>
<button
type="button"
onClick={() => onBuild(source)}
disabled={!source.enabled || isBuilding || isSyncing}
title={t('memorySources.build.title')}
className="inline-flex items-center gap-1 rounded-md border border-primary-300
bg-white px-3 py-1.5 text-xs font-semibold text-primary-600
shadow-sm transition-colors hover:bg-primary-50
disabled:cursor-not-allowed disabled:opacity-50
dark:border-primary-500/30 dark:bg-neutral-900 dark:text-primary-400
dark:hover:bg-primary-500/10
focus:outline-none focus:ring-2 focus:ring-primary-200">
{isBuilding ? <Spinner /> : <BuildIcon />}
{isBuilding ? t('memorySources.build.building') : t('memorySources.build.title')}
</button>
<button
type="button"
onClick={() => onToggle(source)}
Expand Down Expand Up @@ -324,6 +458,14 @@ function relativeTimestamp(epochMs: number | null, t: (k: string) => string): st
return `${days}${t('time.daysAgoSuffix')}`;
}

function sourceTreeScope(source: MemorySourceEntry): string | null {
if (source.kind === 'github_repo' && source.url) {
const m = source.url.match(/github\.com\/([^/]+)\/([^/.]+)/);
if (m) return `github:${m[1]}/${m[2]}`;
}
return source.id;
}

function sourceDetail(source: MemorySourceEntry): string | null {
switch (source.kind) {
case 'composio': {
Expand Down Expand Up @@ -379,6 +521,24 @@ function TrashIcon() {
);
}

function BuildIcon() {
return (
<svg
width="12"
height="12"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
strokeWidth="1.8"
strokeLinecap="round"
strokeLinejoin="round"
aria-hidden="true">
<path d="M22 11.08V12a10 10 0 11-5.93-9.14" />
<path d="M22 4L12 14.01l-3-3" />
</svg>
);
}

function SyncIcon() {
return (
<svg
Expand Down
36 changes: 31 additions & 5 deletions app/src/components/intelligence/MemoryWorkspace.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ import { MemoryGraph } from './MemoryGraph';
import { MemorySourcesRegistry } from './MemorySourcesRegistry';
import { MemoryTreeStatusPanel } from './MemoryTreeStatusPanel';
import { ObsidianVaultSection } from './ObsidianVaultSection';
import { SyncAuditPanel } from './SyncAuditPanel';
import { WhatsAppMemorySection } from './WhatsAppMemorySection';

interface MemoryWorkspaceProps {
Expand All @@ -63,13 +64,13 @@ export function MemoryWorkspace({ onToast }: MemoryWorkspaceProps) {
const [resetting, setResetting] = useState(false);
const [mode, setMode] = useState<GraphMode>('tree');

// (Re)load the graph whenever the mode toggle flips. The Memory
// sources panel manages its own polling.
const [graphVersion, setGraphVersion] = useState(0);

// (Re)load the graph whenever the mode toggle flips or tree events arrive.
useEffect(() => {
console.debug('[ui-flow][memory-workspace] graph load: entry mode=%s', mode);
console.debug('[ui-flow][memory-workspace] graph load: entry mode=%s v=%d', mode, graphVersion);
let cancelled = false;
setError(null);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Avoid synchronous state updates inside useEffect

Line 73 calls setError(null) directly in the effect body. This pattern is flagged by the repo’s react-hooks/set-state-in-effect rule.

Based on learnings: “do not perform synchronous setState directly inside useEffect bodies… react-hooks/set-state-in-effect is enforced in this codebase.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/src/components/intelligence/MemoryWorkspace.tsx` at line 73, The
useEffect inside MemoryWorkspace calls setError(null) synchronously which
violates the react-hooks/set-state-in-effect rule; locate the effect where
setError is invoked and change the immediate call to schedule the state update
asynchronously (for example wrap it in queueMicrotask(() => setError(null)) or
setTimeout(() => setError(null), 0)) so the state update is not performed
synchronously in the effect body.

setGraph(null);
void (async () => {
try {
const resp = await memoryTreeGraphExport(mode);
Expand All @@ -90,7 +91,25 @@ export function MemoryWorkspace({ onToast }: MemoryWorkspaceProps) {
return () => {
cancelled = true;
};
}, [mode]);
}, [mode, graphVersion]);

useEffect(() => {
const onTreeDone = () => {
setTimeout(() => setGraphVersion(v => v + 1), 2000);
};
const onSyncDone = (e: Event) => {
const data = (e as CustomEvent).detail as { stage?: string } | null;
if (data?.stage === 'completed') {
setTimeout(() => setGraphVersion(v => v + 1), 3000);
}
};
window.addEventListener('openhuman:memory-tree-completed', onTreeDone);
window.addEventListener('openhuman:memory-sync-stage', onSyncDone);
return () => {
window.removeEventListener('openhuman:memory-tree-completed', onTreeDone);
window.removeEventListener('openhuman:memory-sync-stage', onSyncDone);
};
}, []);
Comment on lines +96 to +112
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Clear scheduled refresh timers in effect cleanup

The effect unregisters listeners but doesn’t clear pending setTimeout callbacks. Late callbacks can still fire after unmount/remount and trigger unnecessary graph reloads.

Suggested fix
  useEffect(() => {
+    const timerIds: number[] = [];
     const onTreeDone = () => {
-      setTimeout(() => setGraphVersion(v => v + 1), 2000);
+      const id = window.setTimeout(() => setGraphVersion(v => v + 1), 2000);
+      timerIds.push(id);
     };
     const onSyncDone = (e: Event) => {
       const data = (e as CustomEvent).detail as { stage?: string } | null;
       if (data?.stage === 'completed') {
-        setTimeout(() => setGraphVersion(v => v + 1), 3000);
+        const id = window.setTimeout(() => setGraphVersion(v => v + 1), 3000);
+        timerIds.push(id);
       }
     };
     window.addEventListener('openhuman:memory-tree-completed', onTreeDone);
     window.addEventListener('openhuman:memory-sync-stage', onSyncDone);
     return () => {
       window.removeEventListener('openhuman:memory-tree-completed', onTreeDone);
       window.removeEventListener('openhuman:memory-sync-stage', onSyncDone);
+      timerIds.forEach(id => window.clearTimeout(id));
     };
   }, []);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/src/components/intelligence/MemoryWorkspace.tsx` around lines 96 - 112,
The effect schedules setTimeouts in onTreeDone and onSyncDone but never clears
them, so save each timeout ID (e.g., treeTimeoutId and syncTimeoutId) when
calling setTimeout inside onTreeDone and onSyncDone, and call clearTimeout on
those IDs in the useEffect cleanup alongside removing the window listeners;
store IDs in variables or refs (use ReturnType<typeof setTimeout> or number) so
they can be cleared and nulled to avoid stale callbacks that call
setGraphVersion after unmount/remount.


const handleWipe = useCallback(async () => {
// Two-step confirm so accidental clicks can't nuke a workspace.
Expand Down Expand Up @@ -296,6 +315,13 @@ export function MemoryWorkspace({ onToast }: MemoryWorkspaceProps) {
) : (
<MemoryGraph nodes={graph.nodes} edges={graph.edges} mode={mode} />
)}

<div className="rounded-lg border border-stone-100 dark:border-neutral-800 bg-white dark:bg-neutral-900 p-4">
<h3 className="mb-2 text-sm font-medium text-stone-700 dark:text-neutral-200">
{t('sync.auditTitle', 'Sync History')}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Remove fallback literal and use a real i18n key path

Line 321 uses a hardcoded fallback ('Sync History'). In app/src, user-visible copy should resolve from translation keys only, with key parity managed in locale files.

Suggested fix
-          {t('sync.auditTitle', 'Sync History')}
+          {t('sync.auditTitle')}
// app/src/lib/i18n/en.ts
+  'sync.auditTitle': 'Sync History',

As per coding guidelines: “Every user-visible string in app/src/** ... must go through useT() ... and be added to app/src/lib/i18n/en.ts”.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/src/components/intelligence/MemoryWorkspace.tsx` at line 321, The JSX
currently passes a hardcoded fallback string to the t(...) call; remove the
literal fallback and replace it with the proper translation key only (e.g.,
t('sync.auditTitle') instead of t('sync.auditTitle', 'Sync History')), then add
the corresponding entry to the English locale file (app/src/lib/i18n/en.ts)
under the same key used by the component so the text is resolved via useT()/t
without a hardcoded fallback; update MemoryWorkspace.tsx where t is invoked and
ensure key parity in the locale file.

</h3>
<SyncAuditPanel />
</div>
</div>
);
}
Expand Down
Loading
Loading