Repository navigation
Building Filee's own document engine: the story so far (with pictures) #891
Replies: 13 comments
Update: what landed since this postSix more pieces are in the PDF compression: fonts subset again, object streams (#895)Compress PDF now shrinks the fonts a PDF embeds down to the glyphs its pages draw. That helps most with fonts embedded whole, and with pages split off a long document. It also packs the file's objects into object streams. Against Ghostscript's
On PDFs that are already compact, Ghostscript stays 2–12 % smaller, because it rewrites their page content more tightly. So Also try Ghostscript stays as an option. The new engine is a setting now (#899, #900)Settings → Engines → Use the Filee renderer (preview), off by default.
macOS: a false alarm (#896)A font test that skipped on macOS turned out to be a test problem, not a rendering one. macOS reports "no outline" for the space character, which has none. The test now runs on macOS too. RTF without Pandoc (#904)RTF → DOCX, HWPX, PDF, HTML, Markdown is built in now.
Checked with RTF that 한글 itself saved, and with Pandoc's. ODT without Pandoc or LibreOffice (#907)ODT is read and written by Filee itself:
Coming next: an EMF / WMF renderer that works on every platform (clip art in HWP files, Word's EMF pictures), and Filee's own HWP 5.0 reader. |
Update: HWP files and EMF / WMF picturesTwo more pieces are in the Filee reads HWP files itself (#909)
What changes:
To check it, every HWP was compared with the HWPX that 한글 saves from the same file, field by field:
Most of the remaining differences are the document title. When a file has no summary information, 한글's HWPX names it after its first line of text; Filee leaves it empty. The rest is one dropped field result, one distribution document (refused on purpose) and two test artefacts. 21 HWPX files written this way were opened in 한글, and all of them open with the same page count as the original HWP. EMF / WMF pictures on every platform (#908)EMF and WMF are the picture formats of HWP clip art and older Word documents. Until now only Windows could draw them, through GDI+. Filee now has its own EMF / WMF renderer, so they work the same on Windows, Linux and macOS:
Left GDI+, right Filee. The metafiles were recorded on Windows with GDI and GDI+: The one visible difference is the thin dotted line at the top right, which comes out as a grey line. That's on the list. Measured with SSIM (1.0 = identical):
Text scores lowest because GDI snaps letters to the pixel grid and Filee doesn't. Not drawn yet: EMF+ only files (made by GDI+ without an EMF fallback). On Windows those still go to GDI+. Coming next: making the Filee renderer the default, and then removing rhwp. |
Update: Markdown → PowerPoint without PandocFilee now writes PPTX itself (#932, in the The new writer takes anything Filee can read as a document: Markdown, HTML, DOCX, ODT, HWP / HWPX, EPUB, and the Pandoc formats when Pandoc is installed. How the slides are splitThe rules are Pandoc's, so a deck you already make with Pandoc splits the same way:
On a slide: bullet and numbered lists keep their levels, tables get a header row and banding, pictures are scaled to fit (never enlarged), and code goes in a monospace box. When a slide is fullText isn't shrunk to fit. It goes on to a new slide with the same title, like PowerPoint's Continue on a new slide. Lists keep their numbering, and long tables repeat their header row. Checks
Not yet
Coming next: Filee's own slide renderer, so PPTX → PDF / PNG no longer needs LibreOffice. |
Update: old e-books and Kindle output without calibreTwo more pieces are in the Reading CHM, PDB, PML, TCR, RB, SNB and OEB (#931)These formats needed calibre. Filee now reads them itself, so they convert to EPUB, PDF, DOCX, ODT, HWPX, PPTX, TXT and Markdown without calibre.
Three real help files on Windows converted cleanly to EPUB and Markdown: 7-Zip's, PuTTY's, and Python 3.10's (9 MB, 974 pages, 954 pictures). A Korean CHM built in code, converted to PDF with neither calibre nor rhwp installed: The missing space under the table is a bug in the Filee renderer, not in the CHM reader. It's on the list for the renderer work. Writing AZW3 and MOBI (#937)Filee now writes Kindle books itself, from the same XHTML it uses for EPUB. A book gets:
Every book Filee writes is read back with Filee's own MOBI reader and compared with the EPUB of the same book.
These AZW3 and MOBI files haven't been opened in Kindle Previewer or on a Kindle yet. If you try one, a report in this thread helps. What calibre is still for
If you rank calibre first in the engine order, it still writes AZW3 and MOBI. Coming next: the built-in slide renderer (PPTX → PDF / PNG without LibreOffice). |
Update: Unhwp, H2Orestart and Java are goneThree more pieces are in the HWP / HWPX → TXT, Markdown, HTML with Filee's own code (#929)These conversions used Unhwp, a native library. Filee now reads the file itself and writes the text with the same writers it uses for every other document. Because it's all managed code, it works the same on Windows ARM64 and Linux arm64, where Unhwp needed its own builds. What's better than before:
Measured on 30 documents built in code, before Unhwp was removed:
The last 3 differ on purpose: Filee's TXT leaves out headers, footers, page numbers and tracked deletions, and puts notes at the end. LibreOffice without H2Orestart and Java (#934)LibreOffice used to read HWP through H2Orestart, which needs a Java runtime. Filee now reads HWP itself, so LibreOffice no longer opens HWP files. The one target that still needs another engine is RTF: Filee writes a DOCX first and Pandoc or LibreOffice turns it into RTF.
EMF / WMF inside HWPX on every platform (#938)When Filee writes an HWPX, EMF and WMF pictures are now turned into PNG with Filee's own metafile renderer (#908), on Linux and macOS too. Coming next: Filee's own HWP writer, so HWP output no longer needs rhwp. |
Update: Filee writes HWP files itselfFilee now saves The writer is the reverse of the HWP reader from #909. It turns the same HWPX package 한글 saves into HWP records, using Hancom's published format document. Everything the package holds is carried over, including what Filee's document model doesn't keep:
Where the format document leaves a value open, it was matched against HWP files that 한글 saved from the same HWPX, record by record. A file Filee wrote, opened in 한글 2024 (left) and drawn by Filee's renderer (right). The picture, the text flowing around it and the text boxes sit where 한글 puts them: Checks
Eight more test documents (tables, tabs, line spacing, several scripts) still need opening in 한글. The PC was too busy for 한글's automation to start. That check comes next. rhwp stays as a fallback for now. Coming next: with HWP reading (#909) and writing both built in, what's left before rhwp can go is making the Filee renderer the default for HWP → PDF. |
Update: less Pandoc, less 7-ZipTwo more pieces are in the Seven markup languages without Pandoc (#936, #939, #943, #947)Filee now reads and writes these itself:
Each converts to and from Markdown, HTML and TXT, and to DOCX, ODT, HWPX, PPTX and PDF. With these in, Pandoc is only needed for LaTeX, AsciiDoc, Typst and man pages. A notebook built in code (left) and the PDF Filee made from it, with neither Pandoc nor LaTeX installed: The spacing around code and under the table in that picture has since been fixed in the renderer (#946, #949). Every reader was fuzzed with random and damaged input, and none failed. The fuzzer also caught a regular expression in Textile that took minutes on a line of 73 dashes; that is fixed. Unpacking archives without 7-Zip (#950)Filee now unpacks the common archive formats inside the app instead of starting 7-Zip:
The same safety checks apply to every format: password-protected entries, paths that leave the folder, links, archive bombs and too many entries are refused. If an archive uses something the built-in reader doesn't handle, 7-Zip tries it next. A refused archive is never handed on. Unpacking 100 MB in 100 files, compared with the bundled 7-Zip (lower is better): ZIP, TAR, TAR.GZ and TAR.XZ are as fast as 7-Zip or close to it. Solid 7z and bzip2 are much slower. For solid 7z archives with many files, Filee still hands the archive to 7-Zip when it's installed. 7-Zip is still needed to write 7z, XZ and BZ2, and to read CAB, UDF disc images, DMG, LHA and ARJ. |
Update: the Filee renderer now beats rhwp on HWP → PDFThe Filee renderer, Filee's own layout engine (Use the Filee renderer (preview)), now draws more of what 한글 prints. Fourteen fixes are in the What it draws now
Measured against 한글's own PDFsEach document was compared with the PDF 한글 prints. The columns are pages that match, line breaks that match, and median SSIM (a picture similarity score, 1.0 = identical):
No document got worse. SSIM can't get close to 1.0 against these PDFs. When 한글 exports to PDF, it draws all 함초롬바탕 text in the bold face, even where the document doesn't ask for bold. That happens in 44 of the 89 PDFs. What's nextFilee is now ahead of rhwp on every measure, so the next step is making the Filee renderer the default for HWP and HWPX → PDF. rhwp can be removed after that. Still missing: 덧말 (ruby text), line numbers, overlapping characters (글자 겹치기), text art, and some equations. |
Update: slides, spreadsheets and Word 97–2003 without LibreOfficeThree more pieces are in the Slides to PDF and pictures (#942)With Use the Filee renderer (preview) on, PPTX, PPTM, POTX and PPSX now go to PDF, PNG and JPG with Filee's own slide renderer. The renderer draws:
A deck with something it doesn't draw yet (shadows and other effects, 3D, WordArt warps, charts without a stored picture) still goes to LibreOffice. Filee on the left, LibreOffice on the right: A 10-slide deck takes 4–6 seconds. LibreOffice takes about a minute when it starts with a fresh profile. Spreadsheets printed the way Excel prints them (#951)XLSX, XLS and ODS now go to PDF with Filee's own print engine, which follows Excel's page logic:
On 12 test workbooks, page counts and page breaks matched values worked out by hand in Excel's units. Compared with LibreOffice, most break the same way. Where they differ, LibreOffice measures its default column width and its row heights in a different way from Excel. Spreadsheets → PDF through LibreOffice stays in the settings for anyone who prefers it. Charts, shapes and comments aren't printed yet. Word 97–2003 documents (#953)
Password-protected files are refused with a message. Word 6.0 / 95 and Works files still go to LibreOffice. Filee's renderer (left) and LibreOffice (right), from the same DOC: Comments, tracked changes and grouped shapes aren't read yet. Coming next: PPT, ODP / ODG and Visio readers on top of the slide renderer. |
Update: the Filee renderer is now the default for documents → PDFThree more pieces are in the Caught up with main (#1477)The branch was 911 commits behind
Line spacing now matches rhwp line for line (1.0, was 0.875). Lists went down because Filee's gap after the bullet was already wider than rhwp's, and narrower spaces move the text a little further. In the table, the Korean cell text now fits on one line in both renderers, so the one paragraph that was already off weighs more. Security hardening (#1489)An audit went through every built-in reader and writer and the layout engine, looking for files that could hang Filee, use up its memory or close it. The fixes fall into four kinds:
A file over a limit gets the usual "too large" or "too complex" message. The layout engine changes were checked on 80 documents: every page came out byte for byte the same as before. The tests also turned up a few plain bugs, now fixed:
There are 11 new fuzz targets: CFF fonts, cmap tables, PDF content, equations, RTF, ODT, EMF/WMF, the markup readers and writers, TAR, PML and OEB. They run with fixed seeds in every test run. Longer local runs found no failures:
Renderer quality (#1495)These fixes come from what people reported about converted documents. The pictures below are documents built in code. "Filee before" is the branch just before #1495, "Filee now" is after it. There's no Word or LibreOffice on this PC, so there's no reference column. Each document follows what the reports describe Word doing. Paragraph borders and backgrounds (#510, #280, #1313, #65, #1098). Lines under headings and headers, boxes around one or more paragraphs, shading. Word's boxes sit outside the text; 한글's boxes take the text inside, and a box split by a page break stays open at the break. Markdown quotes get their bar and background. The spaces on the left are too wide too: documents from Word, ODT, RTF and Markdown had every space at half the font size. They're now as wide as the font makes them. Right to left (#202, #216, #620). Word and RTF direction settings are read: paragraphs start on the right, tables and columns are mirrored, and footnote numbers stay inside their brackets. Arabic and Hebrew DOCX and RTF no longer need LibreOffice. Chinese and Japanese (#280, #66). Justified lines spread the space between characters, not only at the few spaces. Word's automatic gap (¼ em) goes between Chinese or Japanese and Latin letters or digits. Lines break between any two characters, with the usual rules: closing punctuation never starts a line, opening punctuation never ends one, and a full stop may hang past the margin as in Word. Korean still breaks between words. Word headers and footers (#1152, #416, #556):
Watermarks and WordArt (#1313). Before, the watermark wasn't drawn. It now sits behind the text, turned, see-through and fitted to its box. Picture watermarks are washed out. Word equations (#355):
Text around floating pictures and shapes. Text now goes on both sides of an object in the middle of a column. It also follows "left only", "right only" and "largest side", the distance Word keeps around the object, and tight wrapping around a shape's outline. 한글's text flow settings are read too. Spreadsheet printing (#1315). Rotated, flipped and cropped pictures print turned, mirrored and cropped, and simple shapes are drawn. A sheet with charts, form controls or embedded objects goes to LibreOffice when it's installed. Without it, the sheet is printed and a notice names what was left out. Markdown. Column alignment in pipe tables is kept, and line breaks survive in Markdown output (#1416). Reported cases as tests. 41 problems from issue threads were rebuilt in code as regression tests (no reporters' files). When they were written, 13 of them failed on the renderer and were skipped. All 41 pass now. The Filee renderer is the default (#1495)
Coming next: PPT, ODP / ODG and Visio readers on top of the slide renderer, and Filee's own 7z, XZ, CAB, LHA and ARJ code. They're in review. |
Update: PowerPoint 97–2003, Visio, ODP / ODG and 7z without LibreOffice or 7-ZipOne more batch is in the What no longer needs LibreOffice or 7-Zip
PowerPoint 97–2003 (PPT)Filee reads the binary PPT file itself and turns it into the same slide model as PPTX, so the slide renderer draws it and the PPTX writer saves it. It reads:
Password-protected files are refused with a message. Custom (freeform) shapes, WordArt and OLE objects without a preview picture still send a deck to LibreOffice. Gradients and patterns are drawn in their first colour for now. All 187 preset shapesThe slide renderer drew 107 of PowerPoint's preset shapes. It now draws all 187: callouts, action buttons, bent, curved and circular arrows, ribbons, scrolls, cloud, sun, moon, gears, lightning bolt and the rest. 60 of them are shapes PPT files use, and before this they sent the whole deck to LibreOffice. Twelve of the new shapes in a PPTX built in code, Filee on the left, LibreOffice on the right: The gear teeth are narrower than LibreOffice's, and the cloud callout's small bubbles sit a little differently. The new shapes haven't been compared line by line with the standard's definitions yet. Visio (VSDX)A new Drawings engine turns VSDX into PDF, SVG and PNG, one page per drawing page. It reads masters (and their masters), styles, background pages, layers (hidden and non-printing shapes are left out, as Visio prints), lines, arcs, NURBS curves and polylines, pattern fills, dashed lines, basic arrowheads, text blocks with their character and paragraph formatting, groups and pictures. Old binary OpenDocument presentations and drawings (ODP, OTP, FODP, ODG, FODG)ODP and ODG are read straight into the slide model: masters and backgrounds, footers and slide numbers, bulleted and numbered lists, custom shapes from their formulas, rotations and transforms, gradients, hatches and bitmap fills, tables, groups, and pictures (also the ones embedded in flat XML files). ODP also converts to DOCX, HWPX, ODT, TXT and Markdown. A file with charts, OLE objects, 3D, Fontwork, shadows or form controls still goes to LibreOffice. Connectors are drawn as straight lines, and text isn't shrunk to fit yet. ODP → PPTX stays with LibreOffice. The PPT and ODP pictures have no reference column: they're built in code from the format specs, and LibreOffice doesn't open these hand-built files. Files saved by PowerPoint, Impress or Visio haven't been compared yet. Filee's own 7z, XZ, CAB, LHA and ARJ code
Reading a solid 7z of 100 files (27.9 MB), wall time, best of two:
The PC was busy with other work during these runs, so the times jump around: an earlier run of the same archive took 1.77 s at Writing 8.7 MB in 30 files, single-threaded, CPU time:
At the default level, Filee's 7z is 1.3 % bigger than 7-Zip's and takes about 1.6 times as long. 7-Zip can also use several threads, and Filee uses one. Maximum came out a little bigger than Normal on this data, so its settings still need tuning. Fixed on the way: a 7z that holds only empty files (and the same in CB7 comics) was refused as password-protected. SecurityEvery new reader goes through the same limits as the ones before it. PPT, VSDX, ODP and ODG use the shared document limits (part and package size, XML depth and size, nesting, objects, pages). The archive readers go through the same listing checks (entry count, names, path depth and escapes, compression ratio, free space), the unpack budget and the Mark of the Web. Header counts and dictionary sizes are checked before anything is allocated. A file over a limit gets the usual "too large" or "too complex" message. Each new reader has a fuzz target that runs with fixed seeds in every test run. Longer local runs found no failures:
What still needs LibreOffice or 7-Zip
The engine texts and the LibreOffice download card (in all 22 languages), the README and Coming next: a BZ2 writer, the ARM64 filter for 7z, UDF / IMG / DMG readers, a second round for PPT and VSDX, and an ODP writer. PDF → PS / EPS (#955) is in review. |
Update: 7-Zip is no longer shipped, and PDF → PostScript needs no GhostscriptSeven more pieces are in the What no longer needs which program
With #1508 and #1510, every archive Filee writes is written in-process, and the common disk images are read in-process too. So 7-Zip is no longer shipped (#1519). Setup and the portable packages carry rhwp only. 7-Zip is now an optional engine in Settings → Engines, not ticked by default, for the long tail below. PDF → PS / EPS without Ghostscript (#955)Filee translates the PDF's pages operator by operator into PostScript Level 3. A
Ghostscript 10.08 renders Filee's PS at 300 dpi; it's compared at 100 dpi with Ghostscript's and PDFium's renderings of the PDF. SSIM (1.0 = identical):
Over all 14 samples (17 pages), the lowest score against Ghostscript is 0.9893. In three samples PDFium and Ghostscript already draw the PDF differently (fonts that aren't embedded, tiling patterns, a magnified Type 3 bitmap glyph), so those are only checked against Ghostscript. PS, EPS and AI → PDF, PDF/A and CMYK stay with Ghostscript. Transparency isn't flattened as vectors yet, and EPS files have no preview picture. bzip2 written by Filee (#1508)Filee's own bzip2 writer is written from the format description. It encodes up to 4 blocks at once, and the output is the same byte for byte whatever the number of threads. TAR.BZ2 is written in one pass, without a temporary TAR. 30 MB of mixed data (text, CSV, code-like data and random bytes), wall time, 7-Zip 26.05. The PC was busy with other work during these runs, so the times are rough:
At the default level, Filee's file is 0.1 % bigger and takes about 1.7 times as long; 7-Zip used all 8 logical processors and Filee used 4. On one thread each, Filee took 5.25 s and 7-Zip 7.81 s, but another run of the same 7-Zip command took 5.31 s, so call that even. 7-Zip's The output was unpacked byte for byte by Filee's decoder, SharpCompress and 7-Zip, on 16 kinds of data. Other decoders (the 7-Zip picks its ARM64 or ARM Thumb filter by itself when it packs programs for ARM machines, so 7z files made there no longer need 7-Zip to unpack. The tests reproduce that choice with Windows programs; what 7-Zip picks for Linux and macOS programs wasn't checked. Disk images (#1510)
What the full CI caughtThis PR ran the full CI (macOS arm64 and Intel, Linux x64 and arm64, Windows ARM64), because DMGs made by macOS's
After the fixes all 12 jobs passed, and the macOS arm64 engine tests ran 4,808 tests with 0 failures, the 7-Zip becomes optional (#1519)
PowerPoint 97–2003 and Visio, second round (#1515)PPT now draws freeforms (lines, curves, quarter ellipses, arcs), WordArt without a warp (its font, weight and fill, fitted to the box), linear, centred and multi-colour gradients, and pattern fills in both colours. Shadows are kept when the deck is saved as PPTX. When this was made, LibreOffice didn't open the code-built Visio now resolves "Themed" cells through the drawing's theme (colours, variant colours, fill and line styles, body font), and draws FillGradient gradients, shadows, rounded corners and text backgrounds. Layers that don't print are left out. The theme accent, the variant colour and the shadow match. LibreOffice doesn't draw the themed text colour, the theme's gradient, the FillGradient or the rounded corners. It also puts the text background only behind the letters, where Filee fills the whole text block. Filee's arrowheads are smaller than LibreOffice's; the full arrowhead table isn't done yet. No files saved by PowerPoint or Visio have been compared yet. Still to confirm with them: the PPT gradient angle and the order of its colours, and how Visio's theme is read when cells are missing. ODP writer (#1516)Filee writes OpenDocument presentations from the same slide model the renderer draws:
Read back by Filee, every element's box is within 0.5 pt and the pictures differ by less than 1 % per slide. Opened in LibreOffice 26.2, the test deck and a Markdown deck load and convert to PDF. LibreOffice showed three things the round trip couldn't: gradient angles need their unit, list levels need the label size, and Impress adds the list indent to the paragraph margin. All three are fixed: LibreOffice sets lines with 90 % line spacing a little tighter. The ODP hasn't been opened in PowerPoint or checked against the ODF schema. Shapes aren't marked as placeholders, presets have no adjust handles in Impress, and shadows, charts and animations are approximated. ODP → PPTX stays with LibreOffice, and there's no ODG, OTP or FODP output. Test files LibreOffice opens (#1520)The PPT, ODP and VSDX test builders made files Filee read but LibreOffice refused (PPT, ODP) or drew differently (VSDX). They now write what PowerPoint, Impress and Visio write: a standard compound file and numbered text headers for PPT, a complete manifest for ODP, filled page sheets, style sheets and picture cells for VSDX. New tests open the builders' files in LibreOffice and compare page counts and text with Filee's slides. They run where LibreOffice is installed; CI doesn't have it. The comparison found one real reader bug: Visio's fill patterns were in the wrong order (pattern 7 drew a grid instead of vertical lines). The 15 line and grid patterns are now checked against LibreOffice's Visio import. The dot screens (8–12) and patterns 17, 18 and 24 follow Visio's pattern list, because LibreOffice draws them all alike. They haven't been compared with a file Visio saved. Still different from LibreOffice, not changed: text in a vertically flipped Visio shape (Filee draws it flipped, LibreOffice upright; Visio's own behaviour isn't checked yet), and background pages (LibreOffice also exports them as an extra page). SecurityThe new readers and writers follow the same rules as before:
Each new reader and writer has a fuzz target that runs with fixed seeds in every test run. Longer local runs:
What still needs Ghostscript, LibreOffice or 7-Zip
Not checked yet: HFS+ hard links and device files, real install ISOs and SD card dumps, whether 7-Zip reads exFAT, and the app downloading the real 7-Zip packages (the tests use stand-in files). The engine texts in all 22 languages, the README and Coming next: Filee's own NSIS reader, so Tesseract installs on Windows without 7-Zip (in review), and faster LZMA decoding (in progress). |
Update: NSIS installers, faster LZMA and CRC, Word fixes from your filesFive more pieces are in the Tesseract installs without 7-Zip (#1525)Filee now reads NSIS installers itself, so the Tesseract engine on Windows (an NSIS installer) unpacks without 7-Zip. Since 7-Zip became an optional engine, this removes the "install 7-Zip first" step for Tesseract. The download's SHA-256 check is unchanged. The reader handles solid and non-solid installers with LZMA, NSIS's bzip2, deflate or stored blocks, and NSIS 2 and 3 strings. Faster archives (#1527, #1531)
Word fixes found with the files you attached (#1534)We ran the files from #1152, #1180, #1194, #1313, #993, #1315 and #979 through the Filee renderer and compared them with Word's PDF (or LibreOffice where no Word PDF was attached). None of your files is in the repository; each test builds its document in code.
Caught up with
|



























Uh oh!
There was an error while loading. Please reload this page.
Building Filee's own document engine: where we are
Filee turns HWP / HWPX (and later DOCX and others) into PDF with rhwp today, and sends harder documents to LibreOffice. Both are separate programs. For the past days we have been building an engine of Filee's own that lays out and draws documents in-process, on every platform Filee runs on.
This post tells the story so far, step by step, with pictures. Your comments are very welcome: which documents break for you today, which features matter most, and anything that looks wrong below.
Note
The engine is not in a release yet. It lives on the
builtin-enginesbranch and is tracked on the Filee: built-in engines board. Today's converters are unchanged until the engine is wired in.How to read the pictures
Each picture shows the same document three times:
Most documents come from the test matrix, which is written in code: there are no sample files in the repository. Hangul's PDFs were made by driving Hangul itself through automation. The rest come from files attached to our issues.
1. The groundwork
PDF output and font subsetting (#683)
The first question was how to write PDF. We tried two ways on one mixed page (Korean, Latin, Arabic, Devanagari):
Both gave about the same size, so we kept A. Filee's own subsetter keeps only the glyphs a document uses:
Fonts, shaping and Unicode (#695)
A way to measure (#693)
Before writing layout code, we built the yardstick:
It also showed where rhwp falls short today:
2. Paragraphs and lines (#834, #839)
Line breaking
Following Hangul's own lines
An HWPX that Hangul saved stores every line of every paragraph, and Hangul draws from those lines when it opens the file. Filee now uses them too, when the fonts are installed.
3. Pages, tables and columns (#840, #842, #845)
Page composer
Tables
The two forms from #49 were the hardest test. Against Hangul's own PDFs:
In the instructor card, rhwp draws the consent sentence and the line under it on top of each other. Filee keeps them apart, as Hangul does. (Personal details in the file were already replaced by its author; the signature is blurred here.)
Columns, footnotes and endnotes
4. Floating objects (#847)
Pictures, shapes and text boxes anchored to the paper, the page, the column or the paragraph go where Hangul puts them. On a probe file with one anchor per page, every anchor lands within 0.7 pt of Hangul's.
rhwp loses the wrap and draws two paragraphs on one line. Filee matches Hangul's export of the same file.
5. Equations (#848)
Hangul's equation script is read the way Hangul reads it, then laid out and drawn:
over,sqrt,root … of,LEFT … RIGHT, matrices,cases, sums and integrals;Two exam papers from our issue files, with 488 and 249 equations, were the test bed.
6. Chinese, Japanese and Korean typography (#849)
#280: a table of contents twice as long
In #280's Word document, every page number of the table of contents wrapped onto a line of its own. Word measures tab stops from the margin, Hangul from the paragraph's indent, and that difference pushed the right tab past the margin. Now the numbers stay on their lines at the right margin.
#237: quotes drawn over the text
#237's document showed Chinese quotes drawn over the next character. Filee draws them in place.
7. Right-to-left text and complex scripts (#861)
Bidirectional text
The full Unicode Bidirectional Algorithm (UAX #9) now runs: embeddings, isolates, numbers and brackets. It passes all 861,948 cases of Unicode's two test files.
Shaping
Arabic joining, Indic conjuncts and vowel signs, and Thai, Lao, Khmer and Myanmar marks all come out shaped (#185, #202, #216).
Lines without spaces
Thai, Lao, Khmer and Myanmar put no spaces between words, so lines need a dictionary to know where a word ends. Filee carries ICU's word lists, about 500 KB for the four scripts. Where a document says how to break, Hangul's setting wins:
What's next
Roughly in this order:
.hwpfiles take the same path as HWPX.Not done yet: kashida justification for Arabic, vertical text, CJK punctuation compression, and Word's document grid.
We'd love your input
All reactions