Repository navigation
Nutrient DWS graph body: join wrapped words and CJK lines, tag code fences - #1
Merged
Merged
Conversation
…ences The graph body keeps the page's visual lines, so a word hyphenated across a line break arrives as two words, and a Chinese or Japanese sentence that wraps gets a space at every line break. Both now join: "Regula-" + "tory" becomes "Regulatory", and "健康保险的基" + "本信息" becomes one line with no space. Hangul keeps its line breaks as word breaks. URLs keep their own hyphens, and a hyphen before a capitalised line stays. Code paragraphs now open their fence with the element's codeLanguage when the API reports one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Nutrient DWS graph body keeps the page's visual lines. That breaks text in two ways: a word hyphenated at a line end arrives as two words, and Chinese or Japanese text that wraps gets a space at every line break. This PR joins both in the plain and the styled render, and tags code fences with the language the API reports.
Changes
Regula-+torybecomesRegulatory. A hyphen stays when the next line starts with a capital (Inter-+Provincial), when it belongs to a URL or path, or when it follows a space.健康保险的基+本信息becomes健康保险的基本信息. Hangul is left alone, because Korean separates words with spaces.Codeparagraph opens its fence with the element'scodeLanguage(```python) when the response carries one, and a bare fence otherwise.Testing
tests/parse_bench/inference/providers/parse/test_nutrient_dws_markdown.py: 19 tests pass, 4 of them new (CJK join, Hangul kept, styled CJK heading, code fence language).ruff checkandruff format --checkpass on both files.🤖 Generated with Claude Code