Fix extract-grid embed crashing CAML editor preview

The "Insert Extract Grid" picker in CamlArticleEditor wraps each marker in
a CAML fence so the article renderer can intercept it and mount the
ExtractGridEmbed component. The previous fence used `::: prose`, but
@os-legal/caml's parseBlock has no `case "prose"` — the resulting block
carried `body` instead of `content`, and ProseBlock crashed inside
splitPullquotes(undefined). The crash unmounted the editor modal, which
is why "ArrowDown then Enter inserts the extract-grid component marker"
failed in CT.

Fence type is now `::: oc-component`, dispatched via CamlArticle's
`customBlocks` slot in both CamlArticleEditor and CamlDirectiveRenderer.
The customBlocks renderer hands the fence body back to the existing
[component:...] resolver path so behavior for inline markers is
unchanged.

Also tightened the picker keyboard handler to read activeExtractIndex
from a useRef mirror — back-to-back ArrowDown/Enter keystrokes were able
to fire before React re-ran useCallback, so Enter saw a stale `-1` and
bailed out before insertion.

Address PR #1327 review (Claude bot): captured the SYNC_CONTENT fallback
messageId in a single local so the visible chat entry and the
ChatSourceAtom record share an id when the server omits message_id.

CHANGELOG entries updated.

Author: JSv4

CHANGELOG.md

@@ -53,7 +53,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
-- **`CorpusChat` dropped `SYNC_CONTENT` messages from the visible chat** (Issue #1276, `frontend/src/components/corpuses/CorpusChat.tsx:468-495`): The `SYNC_CONTENT` WebSocket frame is a standalone, non-streaming assistant reply used for synchronous server responses. `ChatTray` (document chat) appends these directly to its `chat` state; the corpus-level chat only forwarded the content to `handleCompleteMessage`, which stores sources in `ChatSourceAtom` but never pushes a message to the visible list. As a result, any `SYNC_CONTENT` the backend sent over the corpus socket rendered nothing. Fixed by mirroring the `ChatTray` pattern — push a new complete assistant message into `chat` before persisting sources/timeline. New regression test in `frontend/tests/CorpusChat.ct.tsx` ("SYNC_CONTENT renders a complete message immediately") pins the behavior.
+- **`CorpusChat` dropped `SYNC_CONTENT` messages from the visible chat** (Issue #1276, `frontend/src/components/corpuses/CorpusChat.tsx:468-505`): The `SYNC_CONTENT` WebSocket frame is a standalone, non-streaming assistant reply used for synchronous server responses. `ChatTray` (document chat) appends these directly to its `chat` state; the corpus-level chat only forwarded the content to `handleCompleteMessage`, which stores sources in `ChatSourceAtom` but never pushes a message to the visible list. As a result, any `SYNC_CONTENT` the backend sent over the corpus socket rendered nothing. Fixed by mirroring the `ChatTray` pattern — push a new complete assistant message into `chat` before persisting sources/timeline. The fallback `crypto.randomUUID()` is also now captured in a single local variable so the visible chat entry and the `ChatSourceAtom` record share the same id when the server omits `message_id`. New regression test in `frontend/tests/CorpusChat.ct.tsx` ("SYNC_CONTENT renders a complete message immediately") pins the behavior.
+- **CAML article preview crashed when inserting an extract grid embed** (`frontend/src/utils/camlComponents.ts`, `frontend/src/hooks/useCamlComponentRenderer.tsx`, `frontend/src/components/corpuses/CamlArticleEditor.tsx`, `frontend/src/components/corpuses/caml/CamlDirectiveRenderer.tsx`): the editor wrapped each newly-inserted `[component:TYPE ...]` marker in a `::: prose` fence, but `@os-legal/caml`'s parser has no `case "prose"` in `parseBlock`, so the resulting block carried `body` instead of `content`. `ProseBlock` then crashed inside `splitPullquotes(undefined)`, which unmounted the entire editor modal and made the "ArrowDown then Enter inserts the extract-grid component marker" CT test fail. Switched the fence to a project-specific `::: oc-component` block and routed it through `CamlArticle`'s `customBlocks` slot, where the marker text is handed back to the existing `[component:...]` resolver. The keyboard handler in `CamlArticleEditor` was also tightened to read the active picker index from a `useRef` mirror so back-to-back ArrowDown/Enter keystrokes don't observe a stale closure value of `-1` and bail out before insertion.
 - **PR #1177 follow-up: CAML extract embed polish** (Issue #1227):
   - **`fullDatacellList` payload now bounded server-side**: `ExtractType.full_datacell_list` accepts optional `limit` / `offset` arguments and the resolver clamps `limit` to `MAX_FULL_DATACELL_LIST_LIMIT` (`opencontractserver/constants/extracts.py`, currently `500`) after permission filtering (`config/graphql/extract_types.py`). `GET_EXTRACT_GRID_EMBED` passes `limit: EXTRACT_GRID_EMBED_CELL_LIMIT` (mirrored at `500` in `frontend/src/assets/configurations/constants.ts`) so pathological extracts no longer transmit thousands of cells just to trigger the too-many-rows guard (`frontend/src/graphql/queries.ts`, `frontend/src/components/extracts/ExtractGridEmbed.tsx`). Full server-side pagination is still tracked in #1204.
   - **`resolveComponentMarker` now receives a stable React key from both call sites**: `useCamlComponentRenderer` and `CamlDirectiveRenderer` pass the marker string as the `key` argument so multiple `[component:...]` blocks in a single article reconcile correctly without React's "missing key prop" warnings (`frontend/src/hooks/useCamlComponentRenderer.tsx`, `frontend/src/components/corpuses/caml/CamlDirectiveRenderer.tsx`). Added regression tests in `frontend/src/utils/__tests__/camlComponents.test.ts`.

frontend/src/components/corpuses/CamlArticleEditor.tsx

@@ -373,6 +373,11 @@ export const CamlArticleEditor: React.FC<CamlArticleEditorProps> = ({
   // Index of the keyboard-focused option within the extract picker dropdown.
   // `-1` means no option is focused (initial state when the dropdown opens).
   const [activeExtractIndex, setActiveExtractIndex] = useState<number>(-1);
+  // Mirror of `activeExtractIndex` in a ref so that the keyboard handler can
+  // read the latest value when consecutive keystrokes (e.g. ArrowDown then
+  // Enter) arrive faster than React schedules a re-render. Without this, the
+  // Enter handler would observe a stale closure value of `-1` and bail out.
+  const activeExtractIndexRef = useRef<number>(-1);
   const textareaRef = useRef<HTMLTextAreaElement>(null);
   const extractPickerRef = useRef<HTMLDivElement>(null);
   const extractPickerTriggerRef = useRef<HTMLButtonElement>(null);
@@ -545,9 +550,11 @@ export const CamlArticleEditor: React.FC<CamlArticleEditorProps> = ({
   }, [showExtractPicker]);
 
   // Reset the keyboard-focused option whenever the picker closes so the next
-  // open starts in a clean state.
+  // open starts in a clean state. The ref mirror is reset alongside state so
+  // the next ArrowDown reads the fresh `-1` value synchronously.
   useEffect(() => {
     if (!showExtractPicker) {
+      activeExtractIndexRef.current = -1;
       setActiveExtractIndex(-1);
     }
   }, [showExtractPicker]);
@@ -600,43 +607,57 @@ export const CamlArticleEditor: React.FC<CamlArticleEditorProps> = ({
       if (!showExtractPicker) return;
       const count = corpusExtracts.length;
 
+      // Helper: update both state (drives render) and ref (drives next
+      // synchronous keystroke). The ref read in the Enter case below would
+      // otherwise observe a stale `-1` if Enter arrives in the same tick as
+      // a preceding ArrowDown.
+      const updateActiveIndex = (next: number) => {
+        activeExtractIndexRef.current = next;
+        setActiveExtractIndex(next);
+      };
+
       switch (event.key) {
         case "Escape":
           event.preventDefault();
           setShowExtractPicker(false);
           extractPickerTriggerRef.current?.focus();
           break;
-        case "ArrowDown":
+        case "ArrowDown": {
           if (count === 0) return;
           event.preventDefault();
-          setActiveExtractIndex((prev) => (prev + 1 >= count ? 0 : prev + 1));
+          const prev = activeExtractIndexRef.current;
+          updateActiveIndex(prev + 1 >= count ? 0 : prev + 1);
           break;
-        case "ArrowUp":
+        }
+        case "ArrowUp": {
           if (count === 0) return;
           event.preventDefault();
           // `prev <= 0` covers both `0` (first item → wrap to last) and `-1`
           // (no item focused → jump to last). This is intentional per WAI-ARIA
           // Authoring Practices for listbox keyboard interaction.
-          setActiveExtractIndex((prev) => (prev <= 0 ? count - 1 : prev - 1));
+          const prev = activeExtractIndexRef.current;
+          updateActiveIndex(prev <= 0 ? count - 1 : prev - 1);
           break;
+        }
         case "Home":
           if (count === 0) return;
           event.preventDefault();
-          setActiveExtractIndex(0);
+          updateActiveIndex(0);
           break;
         case "End":
           if (count === 0) return;
           event.preventDefault();
-          setActiveExtractIndex(count - 1);
+          updateActiveIndex(count - 1);
           break;
         case "Enter": {
           if (count === 0) return;
           // Only act when a menu option is focused — otherwise let the
           // default button behaviour on the trigger toggle the picker.
-          if (activeExtractIndex < 0 || activeExtractIndex >= count) return;
+          const current = activeExtractIndexRef.current;
+          if (current < 0 || current >= count) return;
           event.preventDefault();
           event.stopPropagation();
-          const selected = corpusExtracts[activeExtractIndex];
+          const selected = corpusExtracts[current];
           if (selected) {
             // handleInsertComponent already calls setShowExtractPicker(false)
             // internally, so the picker is closed as part of the insertion.
@@ -649,16 +670,14 @@ export const CamlArticleEditor: React.FC<CamlArticleEditorProps> = ({
           break;
       }
     },
-    [
-      showExtractPicker,
-      corpusExtracts,
-      activeExtractIndex,
-      handleInsertComponent,
-    ]
+    [showExtractPicker, corpusExtracts, handleInsertComponent]
   );
 
   // Markdown renderer with generic component marker interception
-  const renderMarkdownPreview = useCamlComponentRenderer(CAML_COMPONENTS);
+  const {
+    renderMarkdown: renderMarkdownPreview,
+    customBlocks: previewCustomBlocks,
+  } = useCamlComponentRenderer(CAML_COMPONENTS);
 
   const handleClose = () => {
     if (hasChanges) {
@@ -744,7 +763,10 @@ export const CamlArticleEditor: React.FC<CamlArticleEditorProps> = ({
                           key={ext.id}
                           $active={index === activeExtractIndex}
                           aria-selected={false}
-                          onMouseEnter={() => setActiveExtractIndex(index)}
+                          onMouseEnter={() => {
+                            activeExtractIndexRef.current = index;
+                            setActiveExtractIndex(index);
+                          }}
                           onClick={() =>
                             handleInsertComponent("extract-grid", {
                               extractId: ext.id,
@@ -778,6 +800,7 @@ export const CamlArticleEditor: React.FC<CamlArticleEditorProps> = ({
                 <CamlArticle
                   document={parsedDocument}
                   renderMarkdown={renderMarkdownPreview}
+                  customBlocks={previewCustomBlocks}
                 />
               </CamlThemeProvider>
             )}

frontend/src/components/corpuses/CorpusChat.tsx

@@ -470,10 +470,16 @@ export const CorpusChat: React.FC<CorpusChatProps> = ({
             // ASYNC path, it must be appended to `chat` directly or it will never render.
             // No setIsProcessing(false) is needed: ASYNC_START is the only setter for
             // isProcessing(true), and SYNC_CONTENT arrives without a preceding ASYNC_START.
+            //
+            // Capture the message id ONCE so the visible chat entry and the
+            // ChatSourceAtom record agree even if the server omits message_id
+            // (otherwise each `crypto.randomUUID()` call produces a different
+            // value, leaving citations unable to find their parent message).
+            const messageId = data?.message_id ?? crypto.randomUUID();
             setChat((prev) => [
               ...prev,
               {
-                messageId: data?.message_id ?? crypto.randomUUID(),
+                messageId,
                 user: "Assistant",
                 content,
                 timestamp: new Date().toLocaleString(),
@@ -493,7 +499,7 @@ export const CorpusChat: React.FC<CorpusChatProps> = ({
             handleCompleteMessage(
               content,
               sourcesToPass,
-              data?.message_id,
+              messageId,
               undefined,
               timelineToPass
             );

frontend/src/components/corpuses/caml/CamlDirectiveRenderer.tsx

@@ -26,6 +26,7 @@ import {
   DirectiveHandlerContext,
 } from "./directiveRegistry";
 import {
+  OC_COMPONENT_FENCE,
   resolveComponentMarker,
   type CamlComponentRegistry,
 } from "../../../utils/camlComponents";
@@ -187,13 +188,30 @@ export const CamlDirectiveRenderer: React.FC<CamlDirectiveRendererProps> = ({
     componentRegistry,
   ]);
 
+  // Custom block dispatch for the project's `::: oc-component` fence used by
+  // the editor to embed components (e.g. extract grids). The fence wraps a
+  // plain `[component:TYPE ...]` marker, so we delegate to the same
+  // renderMarkdown path that handles inline markers.
+  const customBlocks = useMemo(
+    () => ({
+      [OC_COMPONENT_FENCE]: (block: unknown) => {
+        const body = (
+          (block as { body?: string } | undefined)?.body ?? ""
+        ).trim();
+        return renderMarkdown(body);
+      },
+    }),
+    [renderMarkdown]
+  );
+
   return (
     <CamlThemeProvider>
       <CamlArticle
         document={cleanedDocument}
         stats={stats}
         renderMarkdown={renderMarkdown}
         resolveImageSrc={resolveImageSrc}
+        customBlocks={customBlocks}
       />
     </CamlThemeProvider>
   );

frontend/src/hooks/useCamlComponentRenderer.tsx

@@ -3,31 +3,57 @@
  * for CamlArticle which intercepts `[component:TYPE ...]` markers and
  * renders registered React components in their place.
  *
+ * Also returns a `customBlocks` map that handles the project-specific
+ * `::: oc-component` fence used by the editor's "Insert Extract Grid"
+ * action. The fence is needed because the library's parser does not handle
+ * a `::: prose` fence (block.content is undefined → ProseBlock crashes).
+ *
  * Usage:
  *   const registry = { "extract-grid": ExtractGridEmbed };
- *   const renderMarkdown = useCamlComponentRenderer(registry);
- *   <CamlArticle document={doc} renderMarkdown={renderMarkdown} />
+ *   const { renderMarkdown, customBlocks } = useCamlComponentRenderer(registry);
+ *   <CamlArticle
+ *     document={doc}
+ *     renderMarkdown={renderMarkdown}
+ *     customBlocks={customBlocks}
+ *   />
  */
-import React, { useCallback } from "react";
+import React, { useCallback, useMemo } from "react";
 import { MarkdownMessageRenderer } from "../components/threads/MarkdownMessageRenderer";
 import { ErrorBoundary } from "../components/widgets/ErrorBoundary";
 import { ComponentEmbedErrorFallback } from "../components/widgets/ComponentEmbedErrorFallback";
-import { resolveComponentMarker } from "../utils/camlComponents";
+import {
+  OC_COMPONENT_FENCE,
+  resolveComponentMarker,
+} from "../utils/camlComponents";
 export type { CamlComponentRegistry } from "../utils/camlComponents";
 
+interface OcComponentBlock {
+  type: string;
+  body?: string;
+  attrs?: Record<string, string>;
+}
+
+export interface CamlComponentRendererBindings {
+  /** Pass to `<CamlArticle renderMarkdown={...}>`. */
+  renderMarkdown: (md: string) => React.ReactNode;
+  /** Pass to `<CamlArticle customBlocks={...}>`. */
+  customBlocks: Record<string, (block: unknown) => React.ReactNode>;
+}
+
 /**
- * Returns a stable `renderMarkdown` callback that checks each prose block
- * for a `[component:TYPE ...]` marker. If it matches a registered component,
- * renders that component with the parsed props; otherwise falls back to the
- * standard markdown renderer.
+ * Returns the `renderMarkdown` and `customBlocks` callbacks needed by
+ * `CamlArticle` to resolve `[component:TYPE ...]` markers. `renderMarkdown`
+ * intercepts standalone markers in prose blocks (legacy / inline use); the
+ * `oc-component` custom block intercepts markers wrapped in the project's
+ * dedicated fence.
  */
 export function useCamlComponentRenderer(
   registry: Record<
     string,
     React.ComponentType<Record<string, string | undefined>>
   >
-): (md: string) => React.ReactNode {
-  return useCallback(
+): CamlComponentRendererBindings {
+  const renderMarkdown = useCallback(
     (md: string) => {
       // Use the marker string as the React key so that multiple
       // `[component:...]` blocks in a single article each get a stable,
@@ -48,4 +74,16 @@ export function useCamlComponentRenderer(
     },
     [registry]
   );
+
+  const customBlocks = useMemo(
+    () => ({
+      [OC_COMPONENT_FENCE]: (block: unknown) => {
+        const body = ((block as OcComponentBlock)?.body ?? "").trim();
+        return renderMarkdown(body);
+      },
+    }),
+    [renderMarkdown]
+  );
+
+  return { renderMarkdown, customBlocks };
 }

frontend/src/utils/__tests__/camlComponents.test.ts

@@ -212,12 +212,12 @@ describe("buildComponentMarker()", () => {
 });
 
 describe("buildComponentProseFence()", () => {
-  it("wraps marker in a CAML prose block fence", () => {
+  it("wraps marker in the project's oc-component fence", () => {
     const result = buildComponentProseFence("extract-grid", {
       extractId: "abc",
     });
     expect(result).toBe(
-      "\n::: prose\n[component:extract-grid extractId=abc]\n:::\n"
+      "\n::: oc-component\n[component:extract-grid extractId=abc]\n:::\n"
     );
   });
 });

frontend/src/utils/camlComponents.ts

@@ -126,14 +126,32 @@ export function buildComponentMarker(
 }
 
 /**
- * Wrap a marker in a CAML prose block fence, ready for insertion into
- * the editor source.
+ * Name of the custom CAML fence used to embed an OpenContracts component.
+ *
+ * The library's parser does not have a dedicated case for `::: prose`, so a
+ * `::: prose` fence ends up as `{type: "prose", body, ...}` *without* a
+ * `content` field — and `ProseBlock` then crashes inside `splitPullquotes`.
+ * Using a project-specific block type sidesteps the missing case: unknown
+ * types fall through to the renderer's `customBlocks` lookup, where we own
+ * the rendering and can simply pass the marker text to our resolver.
+ */
+export const OC_COMPONENT_FENCE = "oc-component";
+
+/**
+ * Wrap a marker in a CAML fence ready for insertion into the editor source.
+ *
+ * The marker text is preserved verbatim inside the fence body so existing
+ * marker parsers (`parseComponentMarker`, `resolveComponentMarker`) keep
+ * working against the same `[component:TYPE ...]` shape.
  */
 export function buildComponentProseFence(
   type: string,
   props: CamlComponentProps
 ): string {
-  return `\n::: prose\n${buildComponentMarker(type, props)}\n:::\n`;
+  return `\n::: ${OC_COMPONENT_FENCE}\n${buildComponentMarker(
+    type,
+    props
+  )}\n:::\n`;
 }
 
 // ---------------------------------------------------------------------------