fix(windows): JSON API stdin decodes UTF-8 payload using legacy code page #21

Summary

util.force_utf8_stdio() reconfigures sys.stdout and sys.stderr to UTF-8, but not sys.stdin. On a Chinese-locale Windows install Python decodes stdin with the legacy ANSI code page (GBK / cp936) plus surrogateescape. Any caller that speaks the JSON-lines API over stdio and writes UTF-8 bytes therefore gets silently mis-decoded text: no exception, no warning, just a query that matches nothing.

Reported by @SunnyRad in dsh-sessionflow PR #5. This is a Core-side defect and is not part of, and not caused by, dsh-sessionflow PR #5.

Environment

• Windows with a Chinese system locale (ANSI code page cp936 / GBK)
• Python without PYTHONUTF8=1 and without PYTHONIOENCODING overriding stdin
• sessionFlow Core at 1.3.1 (verified on the current main, ccfd33b)
• A JSON-lines voyager api caller that writes UTF-8 (the DSH bridge, the VS Code client, or any MCP-style stdio client)

Reproduced in a controlled environment by forcing PYTHONIOENCODING=gbk:surrogateescape (equivalent to what a Chinese Windows console does by default). All data below is synthetic; no real user session content is involved.

Expected / Actual

The failing response echoes the decoded query back, which exposes the misdecode:

{"id": 1, "result": {"query": "浼氳瘽", "count": 0, "results": []}}

会话 in UTF-8 is e4 bc 9a e8 af 9d; decoding those six bytes as GBK yields 浼氳瘽. The query never matches because the wrong characters are being searched for.

With a Chinese repo value the same misdecode is worse: overview returns a silent empty list (the filter matches no garbled row), while search surfaces 'utf-8' codec can't encode character '\udcaf' ... surrogates not allowed from the surrogate-escaped bytes. In both cases 0 results / [] is indistinguishable from "there genuinely is nothing".

Root cause

voyager/util.py:31-56 — force_utf8_stdio():

for stream in (sys.stdout, sys.stderr):
    ...
    reconfigure(encoding="utf-8", errors="replace")

sys.stdin is absent from that loop. voyager/api.py:544 (serve()) reads requests with for line in sys.stdin:, so the JSON payload is decoded with the process's legacy stdin encoding before json.loads ever sees it. The fix that shipped for the stdout side (DSH bridge decoding stdout as UTF-8) covered only the output half of the same transport boundary.

Adding sys.stdin to the reconfigured streams resolves all three observations above (verified: 会话 returns 1 hit, the CJK repo empty list and the surrogate error both disappear).

Impact

• Any UTF-8 JSON-lines api client on a non-UTF-8 Windows locale: CJK / non-ASCII query, repo, and other string params are silently corrupted.
• Read-mostly operations fail quietly as empty results — the worst failure mode, because it looks like an empty index.
• voyager api is the transport used by the VS Code client and stdio-based integrations; the DSH bridge is one such caller.
• CLI argv-based commands are unaffected (Windows passes argv as UTF-16), which is exactly why voyager search 会话 --json works while the equivalent API call does not.

Acceptance criteria for a fix

Keep it minimal and at the process boundary; do not blanket-set an encoding for the whole program without checking every input path.

• stdin, stdout and stderr are all UTF-8 at the api / CLI entry points.
• voyager api round-trips a CJK query and a CJK repo path from UTF-8 stdin bytes.
• Windows GBK code page: no silent misdecode, no surrogate escapes leaking into JSON.
• JSON special characters (quotes, backslashes, embedded =, control characters) still parse.
• Invalid UTF-8 input produces a clear error rather than a traceback or a silently empty result.
• MCP stdio entry point (mcp_server.py, which also calls force_utf8_stdio()) stays compatible.
• stdout/stderr behavior from the existing tests/test_utf8_stdio.py regressions is unchanged.
• Tests must spawn a real subprocess and assert on bytes, not monkeypatch sys.stdin in-process: the bug lives in the child's locale-dependent decoding, which an in-process io.TextIOWrapper double cannot reproduce. Cover both the GBK-misdecode case and a UTF-8 control case.

Scope note

This issue is independent of dsh-sessionflow PR #5. PR #5 was a plugin-side fix to argv construction and argument validation; it neither introduced nor can fix this Core-side stdin decoding. It was found while running PR #5's integration tests against a real Core. Do not fold this into that PR.

No real session content is reproduced here — the reproduction uses a synthetic one-session index.