fix(converter): cap doc_name length to avoid Windows path-length errors - #253
Open
sebastianbraun25 wants to merge 1 commit into
Open
fix(converter): cap doc_name length to avoid Windows path-length errors#253sebastianbraun25 wants to merge 1 commit into
sebastianbraun25 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This PR was created in collaboration between a human and AI: implementation, tests, and
PR text were created by an AI assistant under the guidance and review of the human author.
Problem
openkb addcan fail on Windows withOSError [WinError 206]("The filename or extension is too long") when ingesting a source file with a long filename. The sanitized wiki name (doc_name, derived from the original filename's stem) is used unmodified and unbounded in length as a directory/file name component, and appears twice within the per-add staging tree (.openkb/staging/add-{doc_name}-{uuid8}/wiki/sources/images/{doc_name}/). Combined with an already-nested KB path, the total path length can exceed Windows' ~260-char limit.Root Cause
_sanitize_stem()inopenkb/converter.pyonly replaces disallowed characters, never bounds length.doc_name(its return value) is used unmodified incli.py::_staging_dir_for(staging directory name) andconverter.py::convert_document(wiki/sources/images/{doc_name}/), so an overlong original filename appears twice in the same path.openkb/url_ingest.py's_sanitize_filenamealready caps stems at 80 chars (_MAX_FILENAME_STEM) for URL-derived downloads, but that cap wasn't shared with the local-file ingest path.Solution / Changes
openkb/converter.py:_sanitize_stem()now caps the cleaned stem at_MAX_STEM_LEN = 40chars (conservative, sincedoc_namecan appear twice in the staging path) and appends a deterministic 8-hex-char SHA256 digest of the full cleaned stem when truncated, e.g.bulkQuery_result_...-a1b2c3d4. Hashing the full stem (not just the truncated prefix) avoids silent collisions between different overlong names sharing the same prefix.url_ingest.py's counter-based_unique_path(collisions resolved by an on-disk existence check),converter.py's existing collision handling (_name_taken+resolve_doc_name/resolve_doc_name_from_key) is registry-based, so a deterministic hash suffix (not an incremental counter) is used here._staging_dir_for(cli.py) andresolve_doc_name/resolve_doc_name_from_key(converter.py) already call_sanitize_stem(), so both the staging directory and the images directory are fixed by this single change — no other call sites needed updates.-{sha256(path_key)[:8]}) is unchanged and can stack with the new truncation suffix (max length 40+1+8+1+8 = 58 chars — still well within filesystem limits).tests/test_converter.py: newTestSanitizeStemclass covering truncation + hash-suffix format, determinism (same input → same output), and no silent collision between different overlong stems sharing the same 40-char prefix. Existing tests are unaffected (their stems are well under 40 chars).Issues
Resolves #252