Not sure why TEXT_DEHYPHENATE for Page.search_for() is not working

[kai-36]

From the documentation, I understand that by default, search_for will detect for hyphenated words at the end of a line, and join it with the first word in the next line. However, I'm not sure why it's not working for me.

This is a segment of the output from page.get_text():

If replicated on a large scale,
this would, in theory, enable the
ocean to soak up more of the
planet-warming gas driving cli-
mate change.

When I call page.search_for("climate"), it detects "climate" that appear as a whole in the page, but not the hyphenated one.

The text is extracted from a PDF file, and it is justified, if that matters.

[JorjMcKie]

Please provide the example file.

Just to be clear: The search function should find "climate" even when it occurs as "cli-\nmate" in a line break situation. But it will return two hit rectangles, one containing "cli" and a separate one wrapping "mate". IAW there is no (and cannot be a) "joining" in one rectangle.
The MuPDF-logic behind this looks for a number of possible dashes at line ends for this.
If you however do a naive plain text extraction with the de-hyphenate option the text particles should indeed appear joined. Maybe you try this out to see if the problems recurs.

It should be interesting to see what might go wrong in your case.

[kai-36]

Unfortunately, I cannot share the PDF file here. I tried doing plain text extraction with the de-hyphenate option with this line
page.get_text(flags=pymupdf.TEXT_DEHYPHENATE), but the words are not joined.

[JorjMcKie]

Ok, the problem is not specific to search then. You can share the file via email with personally me (I am contractually bound by NDAs, so confidentiality is guaranteed).
But you can do this and look at the text portions of each block:
page.get_text("blocks", flags=0). Look at the respective block, it should look like (x0, y0, x1, y1, "... ocean to soak up more of the\nplanet-warming gas driving cli-\nmate change.",...). If there is a space after "cli-" and before "\n", then this is the explanation.

[kai-36]

test_hyphen.pdf

I created a dummy PDF file that replicates the problem to share with you. I tried getting the text in block form, and there is no whitespace between "cli-" and "\n".

[JorjMcKie]

Thanks for sharing!
I just determined that our behavior has changed here between versions 1.26.7 and 1.27.1.
I am going to turn this discussion into a bug report.

[kai-36]

Ah so it was a bug. Anyways, thanks for the help!