fio UTF-8 everywhere; fread absorbs pipes; collect banner rides force_inscope_pod

[borisbat]

Two bundled slices, per the CI-batching call.

fio: UTF-8 everywhere

das strings are UTF-8 by convention; every fio boundary now converts explicitly instead of letting Windows interpret the bytes in the ANSI codepage:

• das_to_path / path_to_das wrap all 16 std::filesystem call sites — fixes both directions (UTF-8 names misread on the way in; path::string() throwing system_error for unrepresentable names on the way out). Built on MultiByteToWideChar/WideCharToMultiByte, so invalid input degrades to an empty path / U+FFFD instead of throwing.
• dir / chdir / getcwd / mkdir / rmdir and the error-variants go through the wide CRT; rmdir_rec walks wide; normalizeFileName uses GetFullPathNameW.
• The dir_rec representability skip (7b5de4c) is gone — names always convert now, so files stop being invisible.
• tests/fio/fio_utf8.das pins the surface: 20 subtests over Cyrillic/CJK/emoji names (mkdir, stat, round-trip, dir, dir_rec, path helpers, rename, copy, remove, rmdir, file_size, cwd). 9 were red on Windows before the fix.

fread absorbs pipes; fread_to_eof deleted

fread sized reads from fstat().st_size, which is 0 for pipes — a silent empty read (this killed the cpp text-edit provider through popen_argv). The stat is already fetched, so the detection is one bit-test: regular files keep the exact single-read path, everything else reads to EOF with a growing buffer — the same shape Python readall, C# ReadAllBytes, and Rust fs::read use (st_size is a hint, not a contract). fread_to_eof existed only as the workaround and is deleted; all 19 callers migrate to plain fread and shed their unsafe wraps. tests/fio/popen_argv.das is the pipe pin.

containers: the collect banner rides force_inscope_pod

Fallout follow-up to #3602: the finalize banner lowered erase/clear/shrink drops to per-element delete, instantiating and running finalizer codegen old user code never triggered. Replaced with builtin_collect_local — the no-memset sibling of builtin_collect_local_and_zero — which frees a dropped element's owned heap through the GcPod walk and cannot execute user code by construction (pointers and lambdas are not followed; the container disposes of the slot bytes itself). The gate moves with it: needs_container_finalize rides force_inscope_pod (the policy already governing collect on move-assign and scope exit, default OFF), decoupled from default_init_containers, which stays the construct-half knob. isPodDelete + isSafeToDelete still bound what is collectable.

Verification

• tests/fio/fio_utf8.das: 20/20 (was 9 red); fio dir: 222 passed, 0 failed, 1 skipped.
• Container matrix (6 files incl. the heap-delta pins under options force_inscope_pod): all green.
• Full interpreter suite: 13101 passed, 0 failed. JIT sweep: 13010 passed, 0 failures (one 30.4s local time-budget flag on a linq file — the known class, split tracked separately).
• AOT subset links; das2rst clean (no stubs, no Uncategorized); sphinx-build -W green for HTML and LaTeX.
• Lint + format clean on all 25 changed .das files.

Heads-up for external repos: fread_to_eof is gone — replace with fread (drop the unsafe).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR updates the fio boundary to treat das strings as UTF-8 consistently (not ANSI on Windows), folds the former fread_to_eof behavior into fread so pipes/streams are read correctly, and reworks the container “finalize banner” into a “collect banner” that uses builtin_collect_local and is gated by force_inscope_pod.

Changes:

• Windows fio: route std::filesystem and CRT path operations through explicit UTF-8↔UTF-16 conversion (wide CRT / wide iteration) and add a new UTF-8 surface test suite.
• fio: make fread pipe-safe by reading-to-EOF for non-regular files (and size-0 regulars), delete fread_to_eof, and migrate in-tree callers.
• Containers: introduce builtin_collect_local and switch erase/clear/shrink/pop collection to it; move needs_container_finalize gating to force_inscope_pod.

Reviewed changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
utils/preflight/main.das	Updates popen capture to use new pipe-safe fread.
utils/mcp/tools/grep_usage.das	Uses fread for tool probing and refactors output formatting into a helper.
utils/mcp/tools/cpp_format_file.das	Uses fread for clang-format --version probing.
utils/jobque-timeline/tl_launch.das	Uses fread for popen output capture.
utils/dasHerd/watcher/tests/test_watcher_protocol_integration.das	Uses fread for protocol-test capture.
utils/dasHerd/watcher/tests/test_ptyhost.das	Uses fread for ptyhost launcher capture.
utils/dasHerd/watcher/tests/test_changelist_git_cycle.das	Uses fread for git-cycle capture.
utils/dasHerd/watcher/repository_core.das	Uses fread for git output capture.
utils/dasHerd/chunk1/benchmark_session_host.das	Uses fread for file and subprocess stream draining.
utils/dasHerd/chunk0/git_terminal_probe.das	Uses fread for probe output draining.
utils/common/parallel_workers.das	Uses fread for worker subprocess capture.
tests/lint/test_parallel_equivalence.das	Uses fread for popen output capture in lint test.
tests/language/container_init_traits.das	Pins needs_container_finalize to the force_inscope_pod policy in tests.
tests/language/container_init_off.das	Updates expectations/docs around policy split (construct vs collect).
tests/language/container_finalize.das	Updates banner contract text and enables force_inscope_pod for heap-delta assertions.
tests/fio/popen_argv.das	Pins pipe behavior via fread in popen tests.
tests/fio/fio_utf8.das	New test suite covering UTF-8 filenames across fio APIs (mkdir/stat/dir/dir_rec/path helpers/mutations/cwd).
src/simulate/simulate_gc.cpp	Adds builtin_collect_local (GcPod walk without zeroing) for container slot collection.
src/misc/sysos.cpp	Switches normalizeFileName to wide WinAPI for UTF-8 correctness (but see review comment re: strict UTF-8 validation).
src/builtin/module_builtin_runtime.cpp	Registers builtin_collect_local interop.
src/builtin/module_builtin_fio.cpp	Implements UTF-8 path conversions for filesystem boundaries; makes fread pipe-safe; migrates Windows dir/rmdir_rec and other ops to wide rails; removes fread_to_eof export.
src/ast/ast_infer_type.cpp	Changes needs_container_finalize trait to depend on force_inscope_pod (not default_init_containers).
modules/dasLLAMA/tests/run.das	Migrates pipe drain to fread.
modules/dasLLAMA/performance/profile_common.das	Updates docs/comments + uses fread for capture.
modules/dasLLAMA/benchmarks/setup_lcpp_ref.das	Updates docs/comments + uses fread for capture.
modules/dasImgui/widgets/imgui_playwright.das	Uses fread for host capture/drain to avoid pipe-size issues.
modules/dasImgui/text/imgui_text_edit_provider_cpp.das	Uses fread for compiler/version probing and check output capture.
include/daScript/simulate/aot_builtin.h	Declares builtin_collect_local for AOT surface.
include/daScript/simulate/aot_builtin_fio.h	Removes builtin_fread_to_eof declaration.
doc/source/stdlib/handmade/function-fio-fread_to_eof-0x69d02a1f4419cdb.rst	Removes fread_to_eof stdlib documentation stub.
doc/source/reference/language/options.rst	Documents collect-vs-construct policy split and expands force_inscope_pod semantics.
doc/source/reference/language/generic_programming.rst	Updates needs_container_finalize trait definition to match new gating.
daslib/fio.das	Migrates popen capture helper(s) to fread.
daslib/builtin.das	Switches erase/clear/shrink collection to builtin_collect_local.
CLAUDE.md	Updates “collect banner” documentation to match new behavior/gating.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[borisbat]

Fixed in d7badf1 - MB_ERR_INVALID_CHARS now, matching utf8_file_path_to_wide; invalid input returns empty instead of a replacement-char path.

[Copilot]

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated 1 comment.

[borisbat]

Fixed in 422290c - both telegram example files migrated (the sweep had covered daslib/tests/utils/modules but not examples/); a whole-tree grep confirms zero remaining call sites, and both files dry-run compile clean.

[Copilot]

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated no new comments.

Suppressed comments (3)

modules/dasLLAMA/performance/profile_common.das:385

• This comment still describes the old fread behavior (size-via-seek and empty output on non-seekable pipes). In this PR, fio::fread was updated to read non-regular files/streams to EOF, so the comment is now incorrect.

// Run `cmd`, return its trimmed stdout (first line only when `first_line`). Uses fread (fio's
// pipe-safe reader) — fread(f) whole-file sizes via seek and returns empty on a non-seekable pipe.

daslib/builtin.das:2166

• This comment still calls this a "finalize banner", but the implementation now uses builtin_collect_local (collection via GcPod, no user finalizers). Renaming it here keeps the terminology consistent with the rest of the PR and avoids implying user finalizers may run.

    // the finalize banner: release every value's heap (raw-pointer values carved out)
    static_if (typeinfo needs_container_finalize(type<VT>)) {

modules/dasLLAMA/performance/profile_common.das:349

• This docstring says fread was losing multi‑KB pipe output on Windows; after this PR’s fio fix, fread(f) is now pipe-safe, so that rationale is no longer accurate. Please update the wording to avoid suggesting a still-present bug.

This issue also appears on line 384 of the same file.

//! Capture twin of [[run_and_stream]] — argv spawn (no shell parse, the Windows cmd quoting
//! traps can't bite), stdout+stderr merged by popen_argv, whole output returned trimmed.
//! Reads via the same fgets loop as run_and_stream — fread lost multi-KB pipe output
//! on Windows (zen2: a 95-s onnx child's rows captured as empty; the line loop never has).