Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ flowchart LR
| `chunking.py` | Splits a document into meaning-identifiable units (paragraph, sentence, DOM, conversation-turn) plus embedded-image extraction, in document order |
| `embedding_client.py` | Pluggable text-embedding channel (`Null` default, `OpenAiCompatible` real impl) + `chunked_max_similarity` |
| `adjudication_client.py` | Pluggable LLM-judgment channel (`Null` default, `ContextualOrchestrator` real impl) |
| `image_content.py` | Pluggable vision channel: OCR + object recognition/tagging for embedded images (`Null` default, `OpenAiCompatibleVisionClient` real impl) |
| `image_content.py` | Pluggable vision channel: OCR + object recognition/tagging for embedded images (`Null` default, `OpenAiCompatibleVisionClient` real impl). The product popup (`frontend/src/PostBody.tsx`) renders each `data:image` payload in document order so the buyer sees the picture, not the base64 string; GET does not call the vision client. |
| `tepp_client.py` | TEPP's published `AnalysisRunRequest` wire contract, pluggable transport |
| `reconstruct.py` | The pipeline: group → candidate window → score → fuse → thread |
| `lineage_persistence.py` | Flattens reconstruct trees into `post_lineage_edge` row specs (parent, child, fused_score) |
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,17 @@ All notable changes to this project are documented here. Format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versioning follows
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.86.1] - 2026-08-16

### Changed

- Opening a post or its evidence panel now shows each embedded
`data:image` picture in document order, with the surrounding sentences
as text. The raw base64 string is no longer dumped into the popup.
Remote `http(s)` image URLs stay unloaded. After `make seed`, a post
whose body includes a data-URI image shows the picture; Extract Keyman
or Ask still runs OCR on that image for search.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“Extract Keyman or Ask still runs OCR on that image for search” is not true on the shipped default. Those paths call normalize_post_body with _vision_client(), which is Null without VISION_MODEL and then emits [image: content unavailable]. Extract is also post_admin-gated.

Keep the changelog to what a buyer can do after make seed without inventing a live vision channel.


## [0.86.0] - 2026-08-16

### Added
Expand Down
26 changes: 26 additions & 0 deletions docs/image-content-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,18 @@ picture sat relative to the surrounding paragraphs."
| `chunk_position` | `integer not null` | 0-based index among ALL of this document's chunks (text and image together) -- matches `Chunk.index` from `chunk_by_dom` |
| primary key | `(source_document_id, chunk_position)` | one image slot per position per document |

## Viewer contract (before persistence exists)

The demo popup does not yet read these tables. It splits the live
`post_body` the same way `extract_base64_images` does: each
`data:image/...;base64,...` payload becomes an `<img>` at its original
character offset, and the surrounding HTML is shown as text. A buyer who
opens the post sees the picture that sat between the paragraphs, not the
base64 wall. Remote `src="https://..."` tags are stripped, never fetched.
OCR, caption, and tag search still require the vision client on extract /
Ask (Li et al., 2023; Radford et al., 2021) and, in a real deployment,
the tables below.

## Query shapes this supports

- **"Find images whose extracted text or tags match a search query, then
Expand All @@ -105,3 +117,17 @@ picture sat relative to the surrounding paragraphs."
ON CONFLICT DO NOTHING` before the provider call, or a short-lived
lease row) to close that race; this schema documents the storage
guarantee, not that concurrency control.

## References

Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z.,
& Wei, F. (2023). TrOCR: Transformer-based optical character recognition
with pre-trained models. *Proceedings of the AAAI Conference on Artificial
Intelligence, 37*(11), 13094–13102. https://doi.org/10.1609/aaai.v37i11.26538

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S.,
Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I.
(2021). Learning transferable visual models from natural language
supervision. In M. Meila & T. Zhang (Eds.), *Proceedings of the 38th
International Conference on Machine Learning* (pp. 8748–8763). PMLR.
https://proceedings.mlr.press/v139/radford21a.html
2 changes: 1 addition & 1 deletion frontend/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "frontend",
"private": true,
"version": "0.86.0",
"version": "0.86.1",
"type": "module",
"scripts": {
"dev": "vite",
Expand Down
27 changes: 27 additions & 0 deletions frontend/src/App.css
Original file line number Diff line number Diff line change
Expand Up @@ -108,9 +108,36 @@
}

.post-body {
display: flex;
flex-direction: column;
gap: var(--post-body-gap);
}

.post-body-text {
margin: 0;
white-space: pre-wrap;
}

.post-embedded-image {
margin: 0;
padding: var(--post-image-padding);
border: 1px solid var(--post-image-border);
border-radius: var(--post-image-radius);
background: var(--post-image-bg);
}

.post-embedded-image img {
display: block;
max-width: 100%;
height: auto;
}

.post-embedded-image figcaption {
margin-top: 0.4rem;
font-size: 0.85rem;
color: var(--text);
}

.popup-placeholder {
margin-top: 1.5rem;
padding: 1rem;
Expand Down
20 changes: 19 additions & 1 deletion frontend/src/App.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@
failedReportRun?: boolean;
succeededTeppRun?: boolean;
pendingTeppRun?: boolean;
postBody?: string;
}) {
const statusLabel: Record<string, string> = {
open: "Open",
Expand Down Expand Up @@ -668,7 +669,7 @@
jsonResponse({
post_id: "post-1",
post_title: "Public post",
post_body: "The full body text.",
post_body: options?.postBody ?? "The full body text.",
voc_type_code: "voc",
voc_type_label: "Voice of Customer",
visibility_code: "public",
Expand Down Expand Up @@ -1088,6 +1089,23 @@
await waitFor(() => expect(screen.getByText("The full body text.")).toBeInTheDocument());
});

it("shows an embedded invoice image instead of the raw base64 string", async () => {
const tinyPng =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=";
stubBackend({
postBody: `<p>Quote attached.</p><img src="data:image/png;base64,${tinyPng}" alt=""><p>Please confirm.</p>`,
});
render(<App />);
await userEvent.click(await screen.findByRole("button", { name: "View post: Public post" }));

const image = await screen.findByRole("img", { name: /embedded image at character offset/i });
expect(image).toHaveAttribute("src", `data:image/png;base64,${tinyPng}`);
expect(screen.getByText("Quote attached.")).toBeInTheDocument();
expect(screen.getByText("Please confirm.")).toBeInTheDocument();
expect(screen.getByText(/Extract Keyman or ask a question/)).toBeInTheDocument();
expect(screen.queryByText(new RegExp(tinyPng))).not.toBeInTheDocument();
});

it("fetches and renders the post list, then opens a detail popup on click", async () => {
const fetchMock = stubBackend();

Expand Down Expand Up @@ -1781,7 +1799,7 @@
expect(
await screen.findByRole("heading", { name: "Lineage reconstruction · Pending · Demo Corp" }),
).toBeInTheDocument();
expect(screen.getByText(/has not started yet/)).toBeInTheDocument();

Check failure on line 1802 in frontend/src/App.test.tsx

View workflow job for this annotation

GitHub Actions / Frontend lint, test, build

src/App.test.tsx > App, authenticated > records a pending lineage run and opens the authorized detail

TestingLibraryElementError: Unable to find an element with the text: /has not started yet/. This could be because the text is broken up by multiple elements. In this case, you can provide a function for your text matcher to make your matcher more flexible. Ignored nodes: comments, script, style <body> <div> <main> <header class="app-header" > <h1> LineageWeave </h1> <div> <span> demo.analyst </span> <button> Log out </button> </div> </header> <section class="popup-section lineage-home" > <h2> Calendar </h2> <ul class="ticket-list" > <li class="ticket-list-item" > <button aria-label="Open commitment for: Public post" class="post-list-item" > <span class="ticket-title" > Send Northridge Grid the revised quote </span> <span class="post-badge" > Public post </span> <span class="post-badge" > Open </span> <span class="post-badge" > due 2026-01-12 </span> </button> </li> <li class="ticket-list-item" > <button aria-label="Open commitment for: Specification revision requested" class="post-list-item" > <span class="ticket-title" > Send Westfield Power the revised specification </span> <span class="post-badge" > Specification revision requested </span> <span class="post-badge" > Open </span> <span class="post-badge" > due 2026-01-14 </span> </button> </li> </ul> </section> <section class="popup-section lineage-home" > <div class="lineage-home-header" > <h2> Analysis runs </h2> <button aria-label="Request a lineage reconstruction" class="keyman-select" > Request a lineage reconstruction </button> </div> <ul aria-label="Analysis runs" class="ticket-list" > <li class="ticket-list-item" > <button aria-label="Open analysis run: Lineage reconstruction · Succeeded · Demo Corp" class="post-list-item" > <span class="ticket-title" > Lineage reconstruction · Succeeded · Demo Corp </span> <span class="post-badge" > 3 documents </span> </button> </li> <li class="ticket-list-item" > <button aria-label="Open analysis run: TEPP measurement · Failed · Demo Corp" class="post-list-item" > <span class="ticket-title" > TEPP measurement · Failed · Demo Corp </span> <span class="post-badge" > 3 documents </span> <span class="post-meta" > Open this run to see why it failed, then connect the measurement service and re-
const postCall = fetchMock.mock.calls.find(
(call) => String(call[0]).endsWith("/api/analysis-runs") && call[1]?.method === "POST",
);
Expand Down
5 changes: 3 additions & 2 deletions frontend/src/App.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ import {
type VocEvidence,
} from "./api";
import { LineageDag } from "./LineageDag";
import { PostBody } from "./PostBody";
import { subgraphForPost } from "./lineageLayout";
import "./App.css";

Expand Down Expand Up @@ -119,7 +120,7 @@ function EvidencePanel({
{post && (
<>
<h4>{post.post_title}</h4>
<p className="post-body">{post.post_body}</p>
<PostBody body={post.post_body} />
</>
)}
</div>
Expand Down Expand Up @@ -1245,7 +1246,7 @@ function PostDetailPopup({
{post.visibility_label ?? post.visibility_code} &middot;{" "}
{new Date(post.created_at).toLocaleString()}
</p>
<p className="post-body">{post.post_body}</p>
<PostBody body={post.post_body} />

<section className="popup-section">
<h3>요약 (Summary)</h3>
Expand Down
33 changes: 33 additions & 0 deletions frontend/src/PostBody.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
import { splitPostBody, type PostBodySegment } from "./postBodyDisplay";

function renderSegment(segment: PostBodySegment, index: number) {
switch (segment.kind) {
case "text":
return (
<p key={`post-body-text-${index}`} className="post-body-text">
{segment.text}
</p>
);
case "image":
return (
<figure key={`post-body-image-${index}`} className="post-embedded-image">
<img
src={segment.src}
alt={`Embedded image at character offset ${segment.position}`}
/>
<figcaption>
Image from this post. Extract Keyman or ask a question to read text

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This caption is not an action the current viewer can take.

  • Extract Keyman renders only when canExtract (post_admin) and the orchestrator is up (App.tsx). demo.analyst / post_read never sees that button.
  • Extract and Ask both run normalize_post_body(..., vision_client=_vision_client()). _vision_client() is NullImageContentClient unless VISION_MODEL is set. Null does not OCR; it inserts [image: content unavailable].

So the default stack shows a picture and tells the operator to read text inside it via a path that is either hidden or a missing channel. Reword to what this screen actually does. docs/image-content-schema.md already says OCR still requires the vision client.

inside it.
</figcaption>
Comment thread
cursor[bot] marked this conversation as resolved.
</figure>
);
default: {
const _exhaustive: never = segment;
throw new Error(`unexpected post body segment: ${JSON.stringify(_exhaustive)}`);
}
}
}

export function PostBody({ body }: { body: string }) {
return <div className="post-body">{splitPostBody(body).map(renderSegment)}</div>;
}
5 changes: 5 additions & 0 deletions frontend/src/index.css
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@
--accent-bg: rgba(170, 59, 255, 0.1);
--accent-border: rgba(170, 59, 255, 0.5);
--social-bg: rgba(244, 243, 236, 0.5);
--post-body-gap: 0.75rem;
--post-image-padding: 0.75rem;
--post-image-radius: 8px;
--post-image-border: var(--border);
--post-image-bg: var(--code-bg);
--shadow:
rgba(0, 0, 0, 0.1) 0 10px 15px -3px, rgba(0, 0, 0, 0.05) 0 4px 6px -2px;

Expand Down
74 changes: 74 additions & 0 deletions frontend/src/postBodyDisplay.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
import { describe, expect, it } from "vitest";
import { splitPostBody } from "./postBodyDisplay";

/** 1x1 transparent PNG — the same synthetic fixture the Python vision tests use. */
const TINY_PNG_B64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=";

describe("splitPostBody", () => {
it("leaves a plain-text post unchanged so existing popups keep their wording", () => {
expect(splitPostBody("The full body text.")).toEqual([
{ kind: "text", text: "The full body text." },
]);
});

it("keeps comparison operators that look like broken HTML", () => {
expect(splitPostBody("qty < 50 and price > 10")).toEqual([
{ kind: "text", text: "qty < 50 and price > 10" },
]);
});

it("renders a data-URI image as its own segment and never leaks the raw base64 into text", () => {
const html =
`<p>Quote attached.</p><img src="data:image/png;base64,${TINY_PNG_B64}" alt=""><p>Please confirm.</p>`;
const segments = splitPostBody(html);

expect(segments).toEqual([
{ kind: "text", text: "Quote attached." },
{
kind: "image",
src: `data:image/png;base64,${TINY_PNG_B64}`,
mimeType: "image/png",
position: html.indexOf("<img"),
},
{ kind: "text", text: "Please confirm." },
]);
for (const segment of segments) {
if (segment.kind === "text") {
expect(segment.text).not.toContain(TINY_PNG_B64);
expect(segment.text).not.toContain("data:image");
}
}
});

it("keeps two images in document order when a paragraph sits between them", () => {
const html =
`<img src="data:image/png;base64,${TINY_PNG_B64}"><p>between</p>` +
`<img src="data:image/png;base64,${TINY_PNG_B64}">`;
const segments = splitPostBody(html);
expect(segments.map((segment) => segment.kind)).toEqual(["image", "text", "image"]);
expect(segments[1]).toEqual({ kind: "text", text: "between" });
expect(segments[0]?.kind === "image" && segments[0].position).toBe(0);
expect(segments[2]?.kind === "image" && segments[2].position).toBeGreaterThan(0);
});

it("tells the operator to re-export when the base64 payload is not decodable", () => {
const html = '<img src="data:image/png;base64,A">';
expect(splitPostBody(html)).toEqual([
{
kind: "text",
text: "Embedded image could not be decoded. Re-export the source post and open it again.",
},
]);
});

it("does not turn a remote http img into a loaded image", () => {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This remote case is necessary but too narrow: surrounding <p>See</p>…<p>end</p> keeps segments.length > 0, so it never exercises the empty-fallback at postBodyDisplay.ts:68-70.

Missing fixtures that fail on this merged head:

  • remote-only <img src="https://example.test/invoice.png">
  • data:image/png;charset=utf-8;base64, + the 1x1 PNG
  • unquoted src=
  • not-valid-base64!!! (must not re-dump)
  • AA== (must not be kind: "image" without a paint check)
  • evidence-panel render (App.tsx Evidence panel)

const html = '<p>See</p><img src="https://example.test/invoice.png"><p>end</p>';
const segments = splitPostBody(html);
expect(segments.every((segment) => segment.kind === "text")).toBe(true);
expect(segments.map((segment) => (segment.kind === "text" ? segment.text : "")).join(" ")).toContain(
"See",
);
expect(JSON.stringify(segments)).not.toContain("https://example.test");
});
});
72 changes: 72 additions & 0 deletions frontend/src/postBodyDisplay.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
/**
* Split a raw `post_body` into text and in-place data-URI images.
*
* The popup used to dump the source string, so a buyer who opened a post
* with an embedded invoice saw a base64 wall instead of the picture.
* Only `data:image/...;base64,...` payloads are turned into images —
* remote `http(s)` img tags are stripped, never fetched.
*/

export type PostBodySegment =
| { kind: "text"; text: string }
| { kind: "image"; src: string; mimeType: string; position: number };

const DATA_URI_IMG =
/<img\b[^>]*\bsrc\s*=\s*["']data:(image\/[a-zA-Z0-9.+-]+);base64,([A-Za-z0-9+/=\s]+)["'][^>]*>/gi;
Comment thread
cursor[bot] marked this conversation as resolved.

const HTML_TAG = /<\/?[a-zA-Z][^>]*>/g;

const UNDECODEABLE_IMAGE =
"Embedded image could not be decoded. Re-export the source post and open it again.";

function stripHtmlTags(text: string): string {
return text.replace(HTML_TAG, " ").replace(/\s+/g, " ").trim();
}

function isDecodableBase64(raw: string): boolean {
if (raw.length === 0) {
return false;
}
try {
atob(raw);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

atob is not an image decoder. splitPostBody('<img src="data:image/png;base64,AA==">') returns kind: "image" and the UI mounts a broken <img> whose figcaption still says Extract/Ask will read the text.

Treat “decodable” as “the browser can paint this” (magic bytes, plus onerror → re-export copy). Cover AA== in postBodyDisplay.test.ts — today only A (throws) is tested.

return true;
} catch {
return false;
}
}

function pushText(segments: PostBodySegment[], raw: string): void {
const text = stripHtmlTags(raw);
if (text) {
segments.push({ kind: "text", text });
}
}

export function splitPostBody(body: string): PostBodySegment[] {
const segments: PostBodySegment[] = [];
const pattern = new RegExp(DATA_URI_IMG.source, "gi");
let lastIndex = 0;
let match = pattern.exec(body);
while (match !== null) {
pushText(segments, body.slice(lastIndex, match.index));
const mimeType = match[1];
const rawB64 = match[2].replace(/\s+/g, "");
if (isDecodableBase64(rawB64)) {
segments.push({
kind: "image",
src: `data:${mimeType};base64,${rawB64}`,
mimeType,
position: match.index,
});
} else {
segments.push({ kind: "text", text: UNDECODEABLE_IMAGE });
}
lastIndex = match.index + match[0].length;
match = pattern.exec(body);
}
pushText(segments, body.slice(lastIndex));
if (segments.length === 0) {
return [{ kind: "text", text: body }];

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fallback re-dumps the raw body whenever pushText strips every tag to empty.

I ran splitPostBody from this module. These all return the original source string, including payloads this PR claims to hide or strip:

  • <img src="https://example.test/invoice.png"> — remote URL leaks as text (still not fetched, but not stripped)
  • <img src="data:image/png;charset=utf-8;base64,…"> — valid 1x1 PNG, extra parameter, base64 wall returns
  • <img src=data:image/png;base64,…> — unquoted, same wall
  • <img src="data:image/png;base64,not-valid-base64!!!"> — raw dump, not UNDECODEABLE_IMAGE

Plain text already survives via pushText. Returning body here undoes the buyer fix for tag-only / unmatched data-URIs. Return a strip result or the re-export sentence — never the raw source. The current remote test has surrounding <p>See</p> so it never hits this branch.

}
Comment thread
cursor[bot] marked this conversation as resolved.
return segments;
}
2 changes: 1 addition & 1 deletion lineageweave/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,4 +55,4 @@
"sentence_excerpts",
]

__version__ = "0.86.0"
__version__ = "0.86.1"
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "lineageweave"
version = "0.86.0"
version = "0.86.1"
description = "Reconstructs git-branch-style lineage DAGs from scattered short records using multi-channel score fusion and LLM adjudication."
readme = "README.md"
license = { text = "MIT" }
Expand Down
2 changes: 1 addition & 1 deletion uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading