Lesarten are real words from a dictionary in the shared database, not letter swaps

[MarkusNeusinger]

Owner feedback on the Lesart page (2026-08-30): a reading must be an existing word — „Mnhme" is a letter salad, not a Lesart — and the vocabulary belongs in PostgreSQL like everything else.

What

• GET /lesarten?text=… (public, cached like the other reads) answers the vocabulary words that differ from the guess by look-alike letters alone: same length, every differing position a documented pair or chain (core/lesarten: the look-alike table's connected classes make a bucket key — lesart_key("Muhme") == lesart_key("Mühme") — so a query is one indexed lookup; ranking by summed pair distance, the project's own bank words first on a tie, then shorter/alphabetical; the guess itself never). The table is the Python twin of app/src/lib/lesarten.ts and test_lesarten_core.py pins the two together; the page's 32-char cap is pinned to MAX_TEXT_LEN too.
• Vocabulary in the shared DB: migration 0028 adds lesart_forms (gen, key, word, bank) + lesart_dictionary (the live generation, source, size, hash). tools.lesarten.sync expands the igerman98/frami dictionary (tools/lesarten/expand.py: one affix layer, compound-only and needs-affix stems excluded, letter-only forms — 718 667 forms measured) ∪ the quiz bank (641 words, flagged) and loads them through the admin API generation-wise: open (the same content hash as the live build → 409, nothing to do) → batches of 20 000 words, keys computed server-side → commit switches the live generation and drops the old one; an abort drops the half-loaded one. Nothing is committed to the repo but SOURCE.md + fetch_igerman98.py (pinned commit + SHA256s).
• Licence (GPL 2/3): recorded in data/corpora/igerman98/SOURCE.md, quellen-und-rechte.md §5 and datenablage.md §1 — use without conditions, duties only on redistribution of the list, which a handful of words per query is not; the bytes go into neither the image nor the bundle.
• Page: the cards show real words with every differing letter marked and the swaps captioned („ü statt u · n statt e"), bank words labelled „aus der Wortbank", the dictionary named with its size; loading / unreachable / nothing-close states; lib/lesarten.ts keeps only the table (the local swap generator is gone). llms.txt lists the endpoint; the glossary gets „Lesart".

Verification

• Backend: ruff, pytest 1813 passed (new: test_lesarten_core.py, test_lesarten_expand.py, test_api_lesarten.py — load/read/replace/gate/validation on the harness; test_api_public_surface.py lists the two new public reads). /verify-migrations cannot run here (Docker daemon down) — the Migrations job in CI is the check, as agreed before.
• Frontend: tsc, eslint, vitest 171/171; browser against a scratch proxy that answers /lesarten from the locally expanded dictionary with the same core.lesarten code: „Haus" → Hans · Haut · Hanf · Hais · Häms, „lesen" → lehne · sehen (Wortbank) · tuten …, „xyzq" → the „kein Wort" line; console clean.
• tools.lesarten.sync --dry-run: 718 667 words (641 from the bank) in 1.9 s.

After the merge

The endpoint exists only after the deploy; then the load is one command — ADMIN_TOKEN=… uv run python -m tools.lesarten.sync (≈ 36 batches; I will ask before touching prod). Until then the page says „Das Wörterbuch ist noch nicht geladen".

🤖 Generated with Claude Code

https://claude.ai/code/session_012TVu1uouGhsjKCEVhWBinX

[Copilot]

🟡 Changes recommended

There are correctness issues in the generation-load lifecycle (stale generations not actually dropped as documented) and input validation edge cases, plus future-dated provenance/migration headers that should be corrected before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR changes the Lesarten feature from “generated letter swaps” to “real-word readings” backed by a dictionary stored in the shared PostgreSQL database, adding a public GET /lesarten?text=… endpoint plus an admin-gated generation-switched loader and the tooling/docs/tests around it.

Changes:

• Add core/lesarten (look-alike classes, bucketing key, ranking) and a new API router providing GET /lesarten + admin load endpoints.
• Add migration 0028 and SQLAlchemy repository/models for lesart_forms + lesart_dictionary, with generation switching for atomic vocabulary updates.
• Replace the frontend’s local Lesarten generator with API-backed results and update provenance/docs/changelog accordingly.

File summaries

File	Description
tools/lesarten/sync.py	Admin tool to build and load the vocabulary into the API in batches with generation switching.
tools/lesarten/expand.py	Local dictionary expansion (Hunspell affix layer) to produce word forms for loading.
tools/lesarten/init.py	Package docstring describing the tools’ responsibilities and invariants.
tests/test_lesarten_expand.py	Unit tests for Hunspell expansion behavior and filtering rules.
tests/test_lesarten_core.py	Unit tests for look-alike symmetry, TS↔Python table parity, bucketing and ranking.
tests/test_api_public_surface.py	Marks the new Lesarten GET routes as public surface.
tests/test_api_lesarten.py	API harness tests covering load/read/replace/gating/validation for Lesarten.
docs/reference/werkzeuge.md	Documents the new tools/lesarten workflow and constraints (German internal docs).
docs/reference/quellen-und-rechte.md	Records the licensing/provenance doctrine for copyleft wordlists as server data.
docs/reference/glossar.md	Adds glossary entry for “Lesart” and updates the index.
docs/reference/frontend-stack.md	Updates the route description for /lesen/vergleichen to reflect real-word readings.
docs/reference/datenablage.md	Documents the “server data under copyleft” storage rule and public endpoint constraints.
data/corpora/igerman98/SOURCE.md	Adds provenance record for igerman98 and the rationale for server-side use.
data/corpora/igerman98/fetch_igerman98.py	Fetch script with pinned commit + SHA256 verification for the dictionary bytes.
core/lesarten/init.py	Core logic for look-alike bucketing and ranking of candidate readings.
core/database/repositories.py	Adds LesartRepository for dictionary metadata and form generation switching.
core/database/models.py	Adds ORM models LesartForm and LesartDictionary.
core/database/init.py	Exposes the new Lesart models/repository from the core.database package.
CHANGELOG.md	Adds an Unreleased entry describing the new real-word Lesarten behavior and DB-backed vocabulary.
app/src/sections/vergleichen/VergleichenView.tsx	Switches Lesarten UI from local generation to API-backed readings with loading/error states.
app/src/locales/de/vergleichen.ts	Updates German UI strings for the new semantics (real words, dictionary metadata, states).
app/src/lib/lesarten.ts	Reduces to the LOOKALIKES table (no longer generates readings locally).
app/src/lib/lesarten.test.ts	Updates tests to cover table symmetry and constraints after removing the generator.
app/src/lib/api/types.ts	Adds TypeScript types for GET /lesarten response payloads.
app/src/lib/api/endpoints.ts	Adds getLesarten() endpoint wrapper.
app/public/llms.txt	Documents the new endpoint in the LLM-readable public surface listing.
app/prerender/lesen/vergleichen.html	Updates prerendered static content to match the new wording/semantics.
api/schemas.py	Adds Pydantic schemas for Lesarten read/load endpoints and payloads.
api/routers/lesarten.py	New router implementing public read + admin-gated generation load/commit/drop endpoints.
api/routers/init.py	Registers the new Lesarten router.
api/main.py	Includes the Lesarten router in the FastAPI app.
alembic/versions/0028_lesart_forms.py	Migration creating lesart_forms and lesart_dictionary.

Review details

• Files reviewed: 32/32 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[MarkusNeusinger]

Review round addressed in the last push:

• LesartRepository.begin_generation now sweeps every non-live generation before handing out the next number (test test_an_abandoned_load_is_dropped_by_the_next_begin).
• GET /lesarten answers 422 for text that is blank after trimming.
• rank_readings docstring states the actual tie order: cost, bank first, fewer differing letters, alphabetical.

[Copilot]

🟡 Changes recommended

core/lesarten currently treats typed s as always confusable with f, which conflicts with the codebase’s long-ſ vs word-final round-s shaping and can yield incorrect readings (e.g., “das” bucketing with “daf”).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

tools/lesarten/sync.py:62

• The 409 "already live" fast-path is detected via a broad substring check ("409" in the exception text). Since request_json raises SystemExit with a formatted message that also embeds the response body, this can mis-detect unrelated failures (e.g. a 400 whose detail mentions "409"). Match the status marker instead so only an actual HTTP 409 triggers the no-op path.
  core/lesarten/init.py:45
• The LOOKALIKES graph currently treats typed s as always confusable with f. But shaping distinguishes long-ſ vs round-s: a word-final s shapes to Schluss-s and should not be bucketed with f (core/shaping.py makes s round at the end of a letter run). As-is, lesart_key("das") will share a bucket with "daf", potentially returning false readings for final-s words.

    "t": ("l", "f"),
    "l": ("t",),
    "f": ("s", "h", "t"),
    "h": ("f",),
    "s": ("f",),

core/database/repositories.py:151

• Similarly, dropping non-live generations via gen != gen can devolve into a full table scan. Using two bounded deletes (< gen and > gen) better matches the PK layout (gen-first) and avoids scanning the live generation when it’s the only one present.

            meta.forms = forms
            meta.sha256 = sha256
            meta.updated_at = now
        await self.session.execute(delete(LesartForm).where(LesartForm.gen != gen))
        await self.session.flush()

• Files reviewed: 32/32 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Several user-facing and API-contract issues need tightening (generation protocol enforcement, misleading UI/prerender copy, batch cap safety, and future-dated metadata) before this can be safely merged.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

api/routers/lesarten.py:139

• POST /lesarten/dictionary/generations/{gen}/commit also accepts arbitrary gen values. Without enforcing gen == (active_gen + 1), callers can commit unexpected generations and bypass the intended begin→load→commit flow.

async def commit_generation(
    gen: int, body: LesartGenerationIn, db: AsyncSession = Depends(require_db)
) -> LesartDictionaryOut:
    """Make the generation live and drop every other one."""
    repo = LesartRepository(db)
    if await repo.count_forms(gen) == 0:
        raise HTTPException(status.HTTP_409_CONFLICT, detail=f"generation {gen} holds no forms — nothing to commit")
    meta = await repo.commit_generation(gen, body.source, body.sha256)

api/routers/lesarten.py:155

• DELETE /lesarten/dictionary/generations/{gen} should also follow the single-open-generation protocol; as written, it allows dropping arbitrary non-live generations, which is unnecessary if only active_gen + 1 can exist. Enforcing gen == (active_gen + 1) keeps the protocol tight and predictable.

async def drop_generation(gen: int, db: AsyncSession = Depends(require_db)) -> Response:
    """Abandon a load in progress (the live generation cannot be dropped)."""
    repo = LesartRepository(db)
    meta = await repo.dictionary()
    if meta is not None and gen == meta.active_gen:
        raise HTTPException(status.HTTP_409_CONFLICT, detail=f"generation {gen} is live — commit a new one instead")
    await repo.drop_generation(gen)

• Files reviewed: 32/32 changed files
• Comments generated: 4
• Review effort level: Lite

[MarkusNeusinger]

Both sweeps (begin_generation, commit_generation) now delete by two ranges on the leading PK column — _other_generations(keep) = gen < keep OR gen > keep — so the common case with only the live generation present is two empty index range scans.

[Copilot]

🔵 Needs a closer look

There are confirmed functional issues in the new Lesart loading/query path (DB insert strategy for large batches) and in the frontend retry/error-state handling that should be fixed before approval.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

core/database/repositories.py:134

• add_forms pre-queries existing rows using tuple_(key, word).in_(...). With LesartFormsIn allowing up to 50,000 words, this can exceed Postgres’ bind-parameter limit (2 params per tuple) and fail for larger batches. It also does extra round-trips that can be avoided.

Use a single bulk INSERT with ON CONFLICT DO NOTHING (dedupe in-memory so bank=True wins within the batch).

    async def add_forms(self, gen: int, rows: list[tuple[str, str, bool]]) -> int:
        """Insert (key, word, bank) rows into a generation. Rows are deduplicated
        here so a repeated batch cannot violate the primary key."""
        if not rows:
            return 0
        unique = {(key, word): bank for key, word, bank in rows}
        existing = await self.session.execute(
            select(LesartForm.key, LesartForm.word).where(
                LesartForm.gen == gen, tuple_(LesartForm.key, LesartForm.word).in_(list(unique.keys()))
            )
        )
        present = {(row[0], row[1]) for row in existing.all()}
        payload = [
            {"gen": gen, "key": key, "word": word, "bank": bank}
            for (key, word), bank in unique.items()
            if (key, word) not in present
        ]
        if payload:
            await self.session.execute(LesartForm.__table__.insert(), payload)
        return len(payload)

app/src/sections/vergleichen/VergleichenView.tsx:128

• After a failed /lesarten request, failedText is never cleared when starting a new request for the same text, so the UI can keep showing the error state instead of switching back to the loading spinner while a retry is in flight.

Clear failedText when the effect starts (and optionally on success) so retries show the intended loading state.

  useEffect(() => {
    if (!text) return undefined;
    let cancelled = false;
    getLesarten(text)
      .then((out) => {
        if (!cancelled) setAnswer(out);
      })
      .catch(() => {
        if (!cancelled) setFailedText(text);
      });
    return () => {
      cancelled = true;
    };
  }, [text]);

• Files reviewed: 32/32 changed files
• Comments generated: 0 new
• Review effort level: Lite

[MarkusNeusinger]

Round addressed in the last push:

• /forms and /commit accept only the open generation (live + 1, the number begin hands out) — _require_open; a skipped-ahead number is 409 (test added). DELETE keeps accepting any non-live generation: it is the cleanup path.
• LesartFormsIn.words is capped at 20 000 pairs (two bind parameters per pair against PostgreSQL's 65 535), matching the tool's batch.
• dictionaryMissing now says the page names no words until the vocabulary is loaded.
• The prerendered paragraph describes the look-alike list as the catalogue the words are judged by; app/prerender/lesen/vergleichen.html regenerated.

[Copilot]

🟡 Changes recommended

There is at least one confirmed UI correctness issue (stale dictionary note for non-current input) plus an unresolved concurrency risk in generation opening that should be addressed or explicitly mitigated before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

app/src/sections/vergleichen/VergleichenView.tsx:131

• dictionary is derived from the last successful API response even when it belongs to a different text, so the dictionary note can show stale info while the current query is still loading or has failed. Gate dictionary the same way readings is gated (answer.text === text) so the note only reflects the current input.

  const readings: LesartReadingOut[] | null = answer && answer.text === text ? answer.readings : null;
  const dictionary: LesartDictionaryOut | null | undefined = answer?.dictionary;
  const readingsError = failedText === text && readings === null;

api/routers/lesarten.py:84

• LesartSwapOut already supports populating by field name (populate_by_name=True), so using **{"from": ...} is unnecessary and makes this harder to read/maintain. Pass from_= directly and let Pydantic handle the output aliasing.

            LesartReadingOut(
                word=r.word,
                bank=r.bank,
                cost=r.cost,
                swaps=[LesartSwapOut(index=s.index, **{"from": s.from_}, to=s.to) for s in r.swaps],
            )

• Files reviewed: 33/33 changed files
• Comments generated: 1
• Review effort level: Lite