-
Notifications
You must be signed in to change notification settings - Fork 1
feat: show persisted image-region locations (v2.12.8) #405
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
17e9be1
feat: show persisted image-region locations
seonghobae 04bdb18
fix: open image-region locations and separate tag rows
seonghobae 3528ab4
fix: validate image evidence and restore frontend build
seonghobae b170eb4
fix: tolerate normalized image edge rounding
seonghobae ec79f17
test(a11y): cover blank persisted image captions
seonghobae 9f605a5
fix(a11y): keep image fallback names nonblank
seonghobae 80208ba
chore: nudge CI re-review (opencode-agent's prior REQUEST_CHANGES was…
seonghobae fdd1daa
test: preserve OIDC hash through login redirect
seonghobae 0ac8061
fix: normalize blank image-region evidence
seonghobae 9e62536
test(frontend): isolate application URL state
seonghobae 864e8a8
fix(docs): renumber ADR 0133 to 0155 to avoid PR #258 collision
seonghobae 0b1b1fc
fix(frontend): ignore blank parent image OCR
seonghobae ec62d9f
merge origin/main: bring image-region evidence up to date (v2.12.8 ->…
seonghobae b72e18e
Merge branch 'main' into feat/image-region-locations-main
seonghobae 9e0ae38
Merge remote-tracking branch 'origin/main' into HEAD
seonghobae 4d5731f
Merge remote-tracking branch 'origin/main' into HEAD
seonghobae 7ead76f
chore(changelog): renumber the image-region fragment past 2.12.26
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| ## 2.12.27 — Reader-visible image region locations | ||
|
|
||
| - Show persisted image-region bounding ranges beside captions, OCR, and tags. | ||
| - Keep the existing buyer-safe image evidence boundary and hide internal LLM | ||
| guidance. | ||
| - Treat whitespace-only persisted image captions as missing so informative | ||
| source images keep the localized `Embedded image` text alternative instead | ||
| of a blank accessible name. | ||
| - Add the location label to the existing five-locale catalog. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| # ADR 0155: Show persisted image-region locations to buyers | ||
|
|
||
| - Status: Accepted | ||
| - Date: 2026-08-22 | ||
| - Depends on: [0067](0067-visual-region-vision-agent.md), [0091](0091-visual-region-embedding-persistence.md), [0110](0110-buyer-image-evidence-rendering.md) | ||
|
|
||
| ## Context | ||
|
|
||
| Normalized visual-region coordinates (`x_ratio`, `y_ratio`, `width_ratio`, | ||
| `height_ratio`) already persist beside captions, OCR, and tags. The buyer | ||
| popup listed those regions as unlabeled evidence, so an operator could not | ||
| tell where on the source image a caption or OCR excerpt came from. Cai, Yu, | ||
| Wen, and Ma (2003) treat a visual block's bounding box as first-class | ||
| evidence; hiding the box after extraction made the stored coordinates | ||
| unusable at the product boundary. | ||
|
|
||
| Stacked PR work already rendered these locations on a non-main branch. This | ||
| record lands the same buyer action on protected `main` without mixing into | ||
| unrelated stacks. | ||
|
|
||
| ## Decision | ||
|
|
||
| - Render each persisted region's axis-aligned box as a percent range next to | ||
| its caption or OCR excerpt: `Region location: left%, top% – right%, bottom%`. | ||
| - Translate the label through the five-locale UI catalog. | ||
| - Omit the location row when any coordinate is non-finite, negative, or greater | ||
| than one, or when `x_ratio + width_ratio` or `y_ratio + height_ratio` exceeds | ||
| the normalized image boundary. Do not invent a box, a pixel overlay, or an | ||
| internal LLM instruction. | ||
| - Treat an all-whitespace persisted image caption as absent at the rendering | ||
| boundary. Keep the existing localized `Embedded image` short text | ||
| alternative rather than exposing a whitespace-only accessible name, and do | ||
| not add an empty visual `figcaption` merely because whitespace was stored. | ||
| - Treat all-whitespace region captions and OCR excerpts as absent too, so they | ||
| cannot suppress the localized `Unknown` evidence fallback. | ||
| - Keep image tags, OCR tables, and the source raster on their existing | ||
| contracts. This is a presentation boundary, not a new VISION call. | ||
|
seonghobae marked this conversation as resolved.
|
||
|
|
||
| ## Consequences | ||
|
|
||
| After opening a post whose image analysis produced regions, a buyer can read | ||
| where each caption or OCR excerpt sits. Informative source images remain | ||
| programmatically identifiable even if the persisted caption is blank, while | ||
| pure whitespace is no longer surfaced as buyer-visible or assistive-text | ||
| evidence. Internal prompts such as `This post is an image` remain hidden. | ||
| Complete overlay rendering and authorized-corpus quality remain later slices. | ||
|
|
||
| ## References | ||
|
|
||
| Cai, D., Yu, S., Wen, J.-R., & Ma, W.-Y. (2003). *VIPS: A vision-based page | ||
| segmentation algorithm* (Microsoft Research Technical Report MSR-TR-2003-79). | ||
| Microsoft Research. https://www.microsoft.com/en-us/research/publication/vips-a-vision-based-page-segmentation-algorithm/ | ||
|
|
||
| World Wide Web Consortium. (2024). *Web Content Accessibility Guidelines | ||
| (WCAG) 2.2* (W3C Recommendation, December 12, 2024). | ||
| https://www.w3.org/TR/WCAG22/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| import { render, screen } from "@testing-library/react"; | ||
| import { describe, expect, it } from "vitest"; | ||
| import { PostBody } from "./PostBody"; | ||
|
|
||
| const SAFE_PIXEL = | ||
| "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="; | ||
|
|
||
| describe("PostBody image accessibility fallback", () => { | ||
| it("treats a whitespace-only persisted caption as missing", () => { | ||
| const { container } = render( | ||
| <PostBody | ||
| body={`<img src="${SAFE_PIXEL}" />`} | ||
| imageContent={[ | ||
| { | ||
| unit_index: 0, | ||
| mime_type: "image/png", | ||
| status_code: "described", | ||
| extracted_text: " \n ", | ||
| caption: " ", | ||
| tags: [], | ||
| }, | ||
| ]} | ||
| />, | ||
| ); | ||
|
|
||
| expect(screen.getByRole("img", { name: "Embedded image" })).toBeInTheDocument(); | ||
| expect(container.querySelector("figcaption")).toBeNull(); | ||
| expect(container.querySelector(".post-image-text")).toBeNull(); | ||
| }); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| import type { Meta, StoryObj } from "@storybook/react-vite"; | ||
| import { PostBody } from "./PostBody"; | ||
| import diagramSrc from "./fixtures/synthetic-process-diagram.png?inline"; | ||
|
|
||
| const meta = { | ||
| title: "Evidence/PostBody", | ||
| component: PostBody, | ||
| } satisfies Meta<typeof PostBody>; | ||
|
|
||
| export default meta; | ||
|
|
||
| type Story = StoryObj<typeof meta>; | ||
|
|
||
| export const ImageRegionLocations: Story = { | ||
| args: { | ||
| body: `<img src="${diagramSrc}" alt="" />`, | ||
| imageContent: [ | ||
| { | ||
| unit_index: 0, | ||
| mime_type: "image/png", | ||
| status_code: "described", | ||
| extracted_text: "Demo Corp process diagram", | ||
| caption: "A process diagram", | ||
| tags: ["diagram", "process"], | ||
| regions: [ | ||
| { | ||
| region_index: 0, | ||
| x_ratio: 0.1, | ||
| y_ratio: 0.2, | ||
| width_ratio: 0.3, | ||
| height_ratio: 0.4, | ||
| status_code: "described", | ||
| extracted_text: "Title block", | ||
| caption: "Title block", | ||
| tags: ["title"], | ||
| }, | ||
| { | ||
| region_index: 1, | ||
| x_ratio: 0.5, | ||
| y_ratio: 0.18, | ||
| width_ratio: 0.4, | ||
| height_ratio: 0.28, | ||
| status_code: "described", | ||
| extracted_text: "1. Capture 2. Fuse 3. Land", | ||
| caption: "Process steps", | ||
| tags: ["steps"], | ||
| }, | ||
| { | ||
| region_index: 2, | ||
| x_ratio: 0.5, | ||
| y_ratio: 0.55, | ||
| width_ratio: 0.4, | ||
| height_ratio: 0.3, | ||
| status_code: "described", | ||
| extracted_text: "Synthetic fixture only", | ||
| caption: "Notes", | ||
| tags: ["notes"], | ||
| }, | ||
| ], | ||
| }, | ||
| ], | ||
| }, | ||
| }; | ||
|
|
||
| export const WhitespaceCaptionFallsBack: Story = { | ||
| args: { | ||
| body: `<img src="${diagramSrc}" alt="" />`, | ||
| imageContent: [ | ||
| { | ||
| unit_index: 0, | ||
| mime_type: "image/png", | ||
| status_code: "described", | ||
| extracted_text: null, | ||
| caption: " ", | ||
| tags: [], | ||
| }, | ||
| ], | ||
| }, | ||
| }; |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.