Fix canonical IDs with language-aware ASCII transliteration

.github/workflows/ci.yml

@@ -15,7 +15,7 @@ permissions:
 
 jobs:
   slice-tests:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
     strategy:
       fail-fast: false
       matrix:
@@ -51,6 +51,13 @@ jobs:
       - name: Install Poetry
         run: ~/.local/bin/pipx install poetry==2.2.1
 
+      - name: Install ICU 74.2 development libraries
+        if: matrix.slice == 'kg-build'
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libicu-dev pkg-config g++
+          test "$(pkg-config --modversion icu-i18n)" = "74.2"
+
       - name: Install slice dependencies
         run: poetry install --no-interaction --extras "${{ matrix.slice }}"
 
@@ -65,7 +72,7 @@ jobs:
         run: poetry run python tests/tools/run_slice_tests.py "${{ matrix.slice }}" -q
 
   test:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
 
     steps:
       - name: Checkout repository
@@ -86,6 +93,12 @@ jobs:
       - name: Install Poetry
         run: ~/.local/bin/pipx install poetry==2.2.1
 
+      - name: Install ICU 74.2 development libraries
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libicu-dev pkg-config g++
+          test "$(pkg-config --modversion icu-i18n)" = "74.2"
+
       - name: Install dependencies
         run: poetry install --no-interaction --extras "all"
 

CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## Unreleased
+
+### Changed
+
+- KG build canonical IDs use language-aware ASCII transliteration from
+  `account.language`, with stable source hashes for non-ASCII names without URLs.
+  German umlauts use `ae`/`oe`/`ue`; unsupported readings use a hashed fallback.
+  Generated IDs can change; existing root lookup mappings remain authoritative.
+- The `kg-build` and `all` extras require PyICU 2.16.2 and native ICU 74.2 for
+  reproducible transliteration. See `docs/packaging_slices_v7.md` for setup.
+
 ## 8.4.3 - 2026-08-19
 
 ### Fixed

README.md

@@ -31,6 +31,12 @@ pip install "wordlift-sdk[all]"
 
 Requires Python 3.10–3.14.
 
+The `kg-build` and `all` extras also require native ICU 74.2 development
+libraries and a C++ compiler to build PyICU. See the
+[installation instructions](docs/packaging_slices_v7.md#native-icu-for-kg-build).
+Canonical IDs use account-language-aware ASCII transliteration; see the
+[ID policy and migration notes](docs/canonical_id_policy.md#language-aware-ascii-identifiers).
+
 `wordlift-sdk` v7 uses a lean base package plus optional extras. The import
 namespace remains `wordlift_sdk.*`; feature packages load lazily and raise an
 install hint if you access an export without the matching extra installed.

docs/canonical_id_policy.md

@@ -87,3 +87,45 @@ Behavior:
 - duplicate URL rows in dataframe lookup resolve to the shortest IRI path depth
   (tie-break: shorter full IRI, then first row order)
 - lookup misses fall back to normal canonical ID generation
+
+## Language-aware ASCII Identifiers
+
+The canonical generator and `IdAllocator` share ASCII slug normalization.
+Cloud callbacks and both postprocessor worker modes derive language from
+`context.account.language`; direct callers can pass the optional `language`
+argument. Language tags are case-insensitive and accept regional forms such
+as `de-DE` or `de_DE`. Missing language uses Latin accent transliteration only.
+
+- German applies `ä → ae`, `ö → oe`, `ü → ue`, `ß → ss` (including capitals).
+  Other languages retain the generic Latin behavior, such as `ü → u`.
+- Russian, Ukrainian, Greek, Arabic, Hebrew, Korean Hangul, and Hindi use
+  explicit ICU romanization routes. Mandarin Chinese uses `Han-Latin`.
+- Japanese transliterates hiragana and katakana. Kanji, Cantonese, unknown
+  scripts, and scripts outside the selected language route do not receive a
+  guessed pronunciation. Unsupported characters are omitted from the readable
+  slug; an empty result uses `thing`.
+- Inputs are normalized to NFC, so canonically equivalent composed and
+  decomposed text generates the same identifier. Existing punctuation,
+  separators, ASCII-only behavior, and parent nesting conventions remain.
+
+For non-ASCII names without a URL, the readable slug is suffixed with the full
+SHA-256 digest of the NFC-normalized, stripped, lowercase original name
+(before transliteration), even when romanization succeeds. This prevents distinct
+names with the same romanization from merging
+across separate callback graphs. Unsupported names use `thing-<digest>`.
+Names with a URL keep the existing URL-hash suffix. Identical names without
+another identity signal retain their existing ambiguity.
+Source-hash identities do not acquire positional sibling suffixes; graph-local
+collision suffixes still keep repeated identical names separate. This prevents
+sorting newly generated IRIs from changing distinct sibling IDs on a later pass.
+
+Existing authoritative root IRI lookups take precedence over generation.
+The fallback pass preserves already canonical dataset IRIs, but explicitly
+handled roots and dependents can be regenerated. Changing account language or
+regenerating an older non-ASCII identifier can create a new entity, including
+when the URL hash is unchanged. This change does not migrate or delete existing
+entities automatically: retain lookup mappings when existing root identity
+must be preserved.
+
+Transliteration uses PyICU 2.16.2 and native ICU 74.2; installation details are
+in [Packaging Slices](packaging_slices_v7.md#native-icu-for-kg-build).

docs/packaging_slices_v7.md

@@ -71,7 +71,7 @@ distribution model:
   - Modules: `wordlift_sdk.kg_build.*`
   - Dependencies: `advertools`, `gql`, `google-auth`, `gspread`, `jinja2`,
     `lxml`, `morph-kgc`, `worph`, `pandas`, `playwright`, `pydantic-core`, `pyshacl`,
-    `python-liquid`, `rdflib`, `requests`, `tomli`, `tqdm`, `trafilatura`
+    `PyICU`, `python-liquid`, `rdflib`, `requests`, `tomli`, `tqdm`, `trafilatura`
   - Notes: this is intentionally broad because `kg_build` composes multiple
     subsystems.
 
@@ -82,6 +82,47 @@ distribution model:
 - `all`
   - Installs every optional dependency declared above.
 
+## Native ICU for kg-build
+
+Only `kg-build` and `all` install `PyICU==2.16.2`. PyICU builds from source and
+requires ICU development headers/libraries, `pkg-config`, and a C++ compiler.
+The canonical transliteration baseline is ICU **74.2**. Pin the native ICU
+version as well as PyICU in deployment images: ICU data changes can otherwise
+change generated IDs. Non-ASCII canonicalization rejects a different linked
+ICU version with an actionable error; ASCII-only normalization is unchanged.
+CI uses Ubuntu 24.04 and checks this version explicitly.
+
+On Ubuntu 24.04:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y libicu-dev pkg-config g++
+test "$(pkg-config --modversion icu-i18n)" = "74.2"
+uv pip install "wordlift-sdk[kg-build]"
+```
+
+On other platforms, provision ICU 74.2 and set `PKG_CONFIG_PATH` to its
+`lib/pkgconfig` directory before installing the extra. A newer system ICU is
+not an equivalent replacement for reproducible canonical IDs. For example,
+with an existing ICU 74.2 installation under `/opt/local` on macOS:
+
+```bash
+export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig"
+test "$(pkg-config --modversion icu-i18n)" = "74.2"
+uv pip install "wordlift-sdk[kg-build]"
+```
+
+If PyICU was already compiled against another ICU, rebuild it after selecting
+the 74.2 `PKG_CONFIG_PATH` (`uv pip install --reinstall --no-cache pyicu==2.16.2`).
+
+Postprocessor interpreters configured separately must install the same extra
+and native ICU version. Check the linked runtime after installation:
+
+```bash
+python -c 'import icu; print(icu.VERSION, icu.ICU_VERSION)'
+# 2.16.2 74.2
+```
+
 ## Boundary Rules
 
 - Package `__init__` modules must stay lazy so importing `wordlift_sdk` or a

pyproject.toml

@@ -29,6 +29,7 @@ gspread = { version = "^6.1.2", optional = true }
 google-auth = { version = "^2.35.0", optional = true }
 tqdm = { version = "^4.67.1", optional = true }
 advertools = { version = "^0.17.1", optional = true }
+pyicu = { version = "2.16.2", optional = true }
 pycountry = { version = "^26.0.0", optional = true }
 python-liquid = { version = "^2.0.1", optional = true }
 jinja2 = { version = "^3.1.6", optional = true }
@@ -117,6 +118,7 @@ kg-build = [
     "tomli",
     "tqdm",
     "trafilatura",
+    "pyicu",
 ]
 legacy = [
     "gql",
@@ -155,6 +157,7 @@ all = [
     "tqdm",
     "trafilatura",
     "twisted",
+    "pyicu",
 ]
 
 [tool.poetry.group.dev.dependencies]

tests/kg_build/postprocessors/processors/test_id_allocator.py

@@ -1,5 +1,8 @@
 from __future__ import annotations
 
+import hashlib
+import pytest
+
 from rdflib import Graph, Literal, RDF, URIRef
 
 import wordlift_sdk.kg_build.postprocessors.processors.id_allocator as id_allocator_module
@@ -107,3 +110,55 @@ def test_swap_iri_and_helpers() -> None:
 
     assert IdAllocator._first_value(g, new, "name") == "A"
     assert IdAllocator._base_from_priority(g, new) == "A"
+
+
+@pytest.mark.parametrize(
+    "name,language,slug",
+    [("Müller", "de", "mueller"), ("Müller", "tr", "muller"), ("東京", "ja", "thing")],
+)
+def test_multilingual_allocator_identity_and_url_hash(name, language, slug) -> None:
+    allocator = IdAllocator("https://data.example.com/dataset", language=language)
+    graph = Graph()
+    digest = hashlib.sha256(name.lower().encode("utf-8")).hexdigest()
+    expected = URIRef(f"https://data.example.com/dataset/things/{slug}-{digest}")
+    assert allocator.new_independent(graph, base_value=name) == expected
+    subject = URIRef("https://example.com/source")
+    graph.add((subject, URIRef("http://schema.org/name"), Literal(name)))
+    assert allocator.assign(graph, subject) == expected
+    assert allocator.assign(graph, expected) == expected
+    url = "https://example.com/page?b=2&a=1#fragment"
+    result = allocator.new_independent(Graph(), base_value=name, url_value=url)
+    assert str(result).endswith(f"/{slug}-{allocator._url_hash(url)}")
+    child = allocator.new_child(Graph(), parent=expected, base_value=name)
+    assert str(child) == f"{expected}/things/{slug}-{digest}"
+    assert (
+        allocator.new_child(
+            Graph(), parent=expected, base_value=name, force_index=True, index=7
+        )
+        == child
+    )
+
+
+@pytest.mark.parametrize("force_index", [False, True])
+def test_duplicate_unicode_assignments_reuse_their_existing_suffix(force_index):
+    allocator = IdAllocator("https://data.example.com/dataset", language="ja")
+    graph = Graph()
+    originals = [URIRef(f"https://example.com/{index}") for index in range(3)]
+    for index, subject in enumerate(originals):
+        graph.add((subject, URIRef("http://schema.org/name"), Literal("東京")))
+        graph.add(
+            (subject, URIRef("http://schema.org/description"), Literal(str(index)))
+        )
+    allocated = [
+        allocator.assign(graph, subject, force_index=force_index, index=index)
+        for index, subject in enumerate(originals, start=1)
+    ]
+    assert len(set(allocated)) == 3
+    before = set(graph)
+    for _ in range(2):
+        for index, subject in reversed(list(enumerate(allocated, start=1))):
+            assert (
+                allocator.assign(graph, subject, force_index=force_index, index=index)
+                == subject
+            )
+        assert set(graph) == before