Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,8 @@ contextual-orchestrator owns model discovery and selection.
flatten transparent pixels onto white for the derived analysis image while
retaining the original asset and provenance. Recognize image DOM/visual
regions before OCR, descriptions, Keyman extraction, or embeddings. Store
region-level evidence; never show an internal LLM instruction such as
region-level evidence and show each region's bounding range beside its
caption and OCR (ADR 0155); never show an internal LLM instruction such as
`This post is an image` to a buyer.

## Source parsing and semantic units
Expand Down
9 changes: 9 additions & 0 deletions CHANGELOG.d/2.12.27-image-region-locations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
## 2.12.27 — Reader-visible image region locations

- Show persisted image-region bounding ranges beside captions, OCR, and tags.
- Keep the existing buyer-safe image evidence boundary and hide internal LLM
guidance.
- Treat whitespace-only persisted image captions as missing so informative
source images keep the localized `Embedded image` text alternative instead
of a blank accessible name.
- Add the location label to the existing five-locale catalog.
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,12 @@ All notable changes to this project are documented here. Format follows

### Added

- Opening a post with persisted image-region evidence now shows each region's
bounding range beside its caption, OCR, and tags (ADR 0155). After
`make seed`, a synthetic process-diagram region reads **Region location:
0%, 0% – 100%, 100%**. Internal LLM instructions stay hidden. Click the
region list to compare that box with the caption, then read the source
image.
- Ontology neighborhoods can continue beyond the bounded SQL source window
with a versioned opaque HMAC cursor and keyset pagination (ADR 0124 / #363).
A missing process secret keeps the truncated-without-cursor contract.
Expand Down
3 changes: 2 additions & 1 deletion docs/adr/0110-buyer-image-evidence-rendering.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,5 @@ placeholder and is not buyer content.

Buyers see useful image evidence without seeing instructions intended for an
LLM. Search and embedding artifacts remain backed by the existing normalized
parent/region tables and contextual-orchestrator boundary.
parent/region tables and contextual-orchestrator boundary. Buyer-visible
region bounding ranges are specified in [0155](0155-buyer-image-region-locations.md).
56 changes: 56 additions & 0 deletions docs/adr/0155-buyer-image-region-locations.md
Comment thread
seonghobae marked this conversation as resolved.
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# ADR 0155: Show persisted image-region locations to buyers

- Status: Accepted
- Date: 2026-08-22
- Depends on: [0067](0067-visual-region-vision-agent.md), [0091](0091-visual-region-embedding-persistence.md), [0110](0110-buyer-image-evidence-rendering.md)

## Context

Normalized visual-region coordinates (`x_ratio`, `y_ratio`, `width_ratio`,
`height_ratio`) already persist beside captions, OCR, and tags. The buyer
popup listed those regions as unlabeled evidence, so an operator could not
tell where on the source image a caption or OCR excerpt came from. Cai, Yu,
Wen, and Ma (2003) treat a visual block's bounding box as first-class
evidence; hiding the box after extraction made the stored coordinates
unusable at the product boundary.

Stacked PR work already rendered these locations on a non-main branch. This
record lands the same buyer action on protected `main` without mixing into
unrelated stacks.

## Decision

- Render each persisted region's axis-aligned box as a percent range next to
its caption or OCR excerpt: `Region location: left%, top% – right%, bottom%`.
- Translate the label through the five-locale UI catalog.
- Omit the location row when any coordinate is non-finite, negative, or greater
than one, or when `x_ratio + width_ratio` or `y_ratio + height_ratio` exceeds
the normalized image boundary. Do not invent a box, a pixel overlay, or an
internal LLM instruction.
- Treat an all-whitespace persisted image caption as absent at the rendering
boundary. Keep the existing localized `Embedded image` short text
alternative rather than exposing a whitespace-only accessible name, and do
not add an empty visual `figcaption` merely because whitespace was stored.
- Treat all-whitespace region captions and OCR excerpts as absent too, so they
cannot suppress the localized `Unknown` evidence fallback.
- Keep image tags, OCR tables, and the source raster on their existing
contracts. This is a presentation boundary, not a new VISION call.
Comment thread
seonghobae marked this conversation as resolved.

## Consequences

After opening a post whose image analysis produced regions, a buyer can read
where each caption or OCR excerpt sits. Informative source images remain
programmatically identifiable even if the persisted caption is blank, while
pure whitespace is no longer surfaced as buyer-visible or assistive-text
evidence. Internal prompts such as `This post is an image` remain hidden.
Complete overlay rendering and authorized-corpus quality remain later slices.

## References

Cai, D., Yu, S., Wen, J.-R., & Ma, W.-Y. (2003). *VIPS: A vision-based page
segmentation algorithm* (Microsoft Research Technical Report MSR-TR-2003-79).
Microsoft Research. https://www.microsoft.com/en-us/research/publication/vips-a-vision-based-page-segmentation-algorithm/

World Wide Web Consortium. (2024). *Web Content Accessibility Guidelines
(WCAG) 2.2* (W3C Recommendation, December 12, 2024).
https://www.w3.org/TR/WCAG22/
1 change: 1 addition & 0 deletions docs/storybook-inventory.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ buyer-facing control you can click before changing product CSS.
| Story | Buyer next action | Token / module |
|---|---|---|
| `Evidence/CitationChip` | Click a cited title to open that source post. | `--color-chip-border`, `--radius-chip`, `CitationChip` |
| `Evidence/PostBody` | Open Image regions and read each bounding range beside its caption; inspect the whitespace-caption state to confirm the image keeps a usable fallback name. | `--text-muted`, `PostBody` |
| `Evidence/AskEvidenceLayerPopup` | Inspect one citation without leaving the answer; close to continue the answer or open the complete source post. Stories cover text/image evidence, no-evidence, missing OCR, null caption, and blank-caption fallback states. | shared popup tokens through `App.css`, `PopupCloseButton`, `AskEvidenceLayerPopup` |
| `AnalysisRun/CutoffKnownBody` | Read the cutoff-known sentence, then compare it with the live body below. | `--color-accent-border`, `--space-panel-block`, `--radius-panel`, `CutoffKnownBody` |
| `Analysis/LineageEntityPicker` | Choose which corp to reconstruct, then click Request a lineage reconstruction. | `--space-control-gap`, `--size-control-min`, `--radius-control`, `LineageEntityPicker` |
Expand Down
16 changes: 16 additions & 0 deletions frontend/src/App.css
Original file line number Diff line number Diff line change
Expand Up @@ -389,6 +389,22 @@
color: var(--text);
}

.post-image-regions li {
margin: 0.45rem 0;
}

.post-image-regions li > span,
.post-image-regions li > small {
display: block;
}

.post-image-region-location {
display: block;
margin-top: 0.2rem;
font-size: 0.8rem;
color: var(--text-muted);
}

.popup-placeholder {
margin-top: 1.5rem;
padding: 1rem;
Expand Down
4 changes: 3 additions & 1 deletion frontend/src/App.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -29,20 +29,22 @@ beforeEach(() => {

afterEach(() => {
vi.unstubAllGlobals();
window.history.replaceState({}, "", "/");
window.sessionStorage.clear();
window.localStorage.clear();
});

describe("App, unauthenticated", () => {
it("shows a login button that starts the real OIDC redirect", async () => {
window.history.replaceState({}, "", "/?post=abc#evidence");
Comment thread
seonghobae marked this conversation as resolved.
render(<App showLabPanels />);
expect(screen.queryByRole("heading", { name: /admin settings/i })).toBeNull();
const button = screen.getByRole("button", { name: /log in/i });
await userEvent.click(button);
expect(signinRedirect).toHaveBeenCalledTimes(1);
expect(signinRedirect).toHaveBeenCalledWith(
expect.objectContaining({
state: expect.objectContaining({ returnUrl: expect.stringMatching(/^\//) }),
state: { returnUrl: "/?post=abc#evidence" },
}),
);
// Persisted as a fallback in case the OIDC state round-trip is dropped
Expand Down
30 changes: 30 additions & 0 deletions frontend/src/PostBody.blank-caption.test.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
import { render, screen } from "@testing-library/react";
import { describe, expect, it } from "vitest";
import { PostBody } from "./PostBody";

const SAFE_PIXEL =
"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=";

describe("PostBody image accessibility fallback", () => {
it("treats a whitespace-only persisted caption as missing", () => {
const { container } = render(
<PostBody
body={`<img src="${SAFE_PIXEL}" />`}
imageContent={[
{
unit_index: 0,
mime_type: "image/png",
status_code: "described",
extracted_text: " \n ",
caption: " ",
tags: [],
},
]}
/>,
);

expect(screen.getByRole("img", { name: "Embedded image" })).toBeInTheDocument();
expect(container.querySelector("figcaption")).toBeNull();
expect(container.querySelector(".post-image-text")).toBeNull();
});
});
79 changes: 79 additions & 0 deletions frontend/src/PostBody.stories.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
import type { Meta, StoryObj } from "@storybook/react-vite";
import { PostBody } from "./PostBody";
import diagramSrc from "./fixtures/synthetic-process-diagram.png?inline";

const meta = {
title: "Evidence/PostBody",
component: PostBody,
} satisfies Meta<typeof PostBody>;

export default meta;

type Story = StoryObj<typeof meta>;

export const ImageRegionLocations: Story = {
args: {
body: `<img src="${diagramSrc}" alt="" />`,
imageContent: [
{
unit_index: 0,
mime_type: "image/png",
status_code: "described",
extracted_text: "Demo Corp process diagram",
caption: "A process diagram",
tags: ["diagram", "process"],
regions: [
{
region_index: 0,
x_ratio: 0.1,
y_ratio: 0.2,
width_ratio: 0.3,
height_ratio: 0.4,
status_code: "described",
extracted_text: "Title block",
caption: "Title block",
tags: ["title"],
},
{
region_index: 1,
x_ratio: 0.5,
y_ratio: 0.18,
width_ratio: 0.4,
height_ratio: 0.28,
status_code: "described",
extracted_text: "1. Capture 2. Fuse 3. Land",
caption: "Process steps",
tags: ["steps"],
},
{
region_index: 2,
x_ratio: 0.5,
y_ratio: 0.55,
width_ratio: 0.4,
height_ratio: 0.3,
status_code: "described",
extracted_text: "Synthetic fixture only",
caption: "Notes",
tags: ["notes"],
},
],
},
],
},
};

export const WhitespaceCaptionFallsBack: Story = {
args: {
body: `<img src="${diagramSrc}" alt="" />`,
imageContent: [
{
unit_index: 0,
mime_type: "image/png",
status_code: "described",
extracted_text: null,
caption: " ",
tags: [],
},
],
},
};
Loading
Loading