fix(filesystem): preserve UTF-8 multibyte characters across chunk boundaries in head/tail

[ishidakei]

Description

read_text_file with head or tail corrupts UTF-8 multibyte characters that straddle the 1024-byte chunk read boundary. The corrupted bytes become U+FFFD (replacement character). Reading the same file in full mode (no head/tail) is unaffected.

Server Details

• Server: filesystem
• Changes to: lib.ts (headFile, tailFile)

Motivation and Context

Both headFile and tailFile read the file as raw byte chunks via fileHandle.read(chunk, 0, size, position) and then call .toString('utf-8') on each chunk independently before concatenating the resulting strings. UTF-8 multi-byte sequences (e.g. CJK characters at 3 bytes each, most non-Latin scripts at 2–4 bytes) that span a chunk boundary are split mid-sequence, so each chunk decodes its partial bytes as U+FFFD and the original character is lost.

Reproduction (default 1024-byte chunk):

• File: 1023 bytes of ASCII, then あ (UTF-8: E3 81 82), then a newline.
• head: 1 returns ASCII followed by ��� instead of あ.

The same mechanism breaks tail symmetrically: any multi-byte character spanning a 1024-byte boundary measured from EOF is replaced with U+FFFD.

This impacts every script outside Basic Latin in practice — Japanese/Chinese/Korean, Cyrillic, Greek, Arabic, accented Latin (German umlauts, French/Spanish accents, etc.), and emoji.

Changes

• headFile: use node:string_decoder StringDecoder so trailing partial bytes from one read are held internally and combined with the leading bytes of the next read. decoder.end() is called on EOF to flush any genuinely truncated sequence.
• tailFile: accumulate raw bytes into a Buffer[] in iteration order and decode the concatenated buffer once at the end. Newline counting is performed at the byte level (safe because 0x0A never appears inside a UTF-8 multi-byte sequence by design). The discarded partial first line absorbs any U+FFFD introduced at the lowest read position.

Behavior on ASCII input is unchanged. Memory characteristics of tailFile are unchanged (chunks accumulated correspond to the same range of bytes the original read).

How Has This Been Tested?

• Added vitest tests in src/filesystem/__tests__/encoding.test.ts using real temp file I/O (no fs mocking) that place 3-byte (あ) and 4-byte (😀) UTF-8 characters at exact 1024-byte chunk boundaries for both head and tail directions. The new tests fail on main and pass with this change.
• Existing lib.test.ts and other suites continue to pass — full local run shows 156 passed across 8 test files.
• Originally observed via Claude.ai with Filesystem MCP reading a Markdown file containing Japanese kanji: full-file reads were clean while read_text_file with head or tail returned U+FFFD at multiples of 1024 bytes from the relevant end.

Breaking Changes

None. The fix only affects previously incorrect output for non-ASCII input.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Checklist

• I have read the MCP Protocol Documentation
• My changes follow MCP security best practices
• I have updated the server's README accordingly (no README change needed — read_text_file already documents UTF-8 support; this fix brings behavior in line with documentation)
• I have tested this with an LLM client
• My code follows the repository's style guidelines
• New and existing tests pass locally
• I have added appropriate error handling
• I have documented all environment variables and configuration options (N/A)