Skip to content

Use language-aware ASCII transliteration for canonical IDs #34

Description

@ziodave

Problem

The IRI_UTF_fix branch deletes non-ASCII letters. Distinct non-Latin names can collapse to one canonical entity, and Latin letters such as ß, ł and ı are lost.

Goal

Generate ASCII identifiers using account.language, shared ICU transliteration, and deterministic identity suffixes.

Scope

  • Shared normalization for allocator and both generator strategies
  • German umlaut rules and explicit supported language routes, with deterministic fallback for unsupported readings
  • Account language propagation through cloud, one-shot and persistent contexts
  • SHA-256 of normalized original non-ASCII names when no URL hash is available
  • PyICU dependency, packaging verification, documentation and regression tests

Non-goals

Automatic migration/deletion of stored entities, universal pronunciation inference, and redesign of existing ASCII-only identity rules.

Acceptance criteria

  • Distinct non-Latin names remain distinct within and across callback graphs
  • German ü becomes ue; Turkish ü does not use German rules
  • Japanese kanji never silently receives Chinese readings
  • Allocator and both generator strategies agree and remain stable on repeated runs
  • Existing URL hashes, root lookup reuse and parent nesting are preserved
  • Language survives both worker modes without cross-job leakage
  • Focused tests, kg-build slice checks and Python 3.14 validation pass

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions