Skip to content

fix(filesystem): preserve UTF-8 across head and tail chunks - #4667

Open
Excelius-Wang wants to merge 1 commit into
modelcontextprotocol:mainfrom
Excelius-Wang:fix/filesystem-utf8-chunk-boundaries
Open

fix(filesystem): preserve UTF-8 across head and tail chunks#4667
Excelius-Wang wants to merge 1 commit into
modelcontextprotocol:mainfrom
Excelius-Wang:fix/filesystem-utf8-chunk-boundaries

Conversation

@Excelius-Wang

Copy link
Copy Markdown

Description

Fixes #4666.

Preserve multi-byte UTF-8 characters when headFile() and tailFile() read across 1024-byte chunk boundaries.

  • Use StringDecoder for forward chunk decoding in headFile().
  • Keep reverse-read chunks as bytes in tailFile() and decode only after enough line separators have been collected.
  • Add regression tests with a CJK character split across the chunk boundary in both directions.

Server Details

  • Server: filesystem
  • Changes to: head_file and tail_file

Motivation and Context

Decoding every chunk independently replaces split UTF-8 sequences with U+FFFD. This corrupts CJK and other multi-byte text returned by the filesystem server.

How Has This Been Tested?

  • npm test — 154 tests passed
  • npm run build
  • Added regression tests for headFile() and tailFile() with split across a 1024-byte boundary

The change was tested at the filesystem operation boundary rather than through an LLM client.

Breaking Changes

None.

Types of changes

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update

Checklist

  • The code follows the repository style
  • New and existing tests pass locally
  • No README update is required because the public interface and configuration are unchanged
  • No environment variables or configuration options were added

Copilot AI balanced review requested due to automatic review settings August 19, 2026 16:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

headFile/tailFile in filesystem server corrupts multi-byte UTF-8 characters at 1024-byte chunk boundaries

2 participants