Skip to content

Split tokenizer state-machine responsibilities - #101

Merged
dv00d00 merged 2 commits into
mainfrom
refactor/split-tokenizer-states
Aug 20, 2026
Merged

Split tokenizer state-machine responsibilities#101
dv00d00 merged 2 commits into
mainfrom
refactor/split-tokenizer-states

Conversation

@dv00d00

@dv00d00 dv00d00 commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Summary

  • split ordinary tag-state handling from tag token lifecycle code
  • isolate the fused tag-tail scanner in its own partial file
  • separate tokenizer configuration, write-loop, and output helpers
  • preserve the tokenizer instruction stream; this is a structural-only refactor

Validation

  • whitespace-insensitive reconstructed token streams match the original Tags and Common files
  • dotnet build AngleSharp.ReadOnlyDom.slnx -c Release --no-restore -m:1 (0 warnings)
  • 179,562 / 179,562 net10.0 tests passed via the TUnit executable
  • UTF-8 token smoke: 57 documents / 38,831,169 bytes, 0 failures
  • UTF-8 DOM projection and QQ scraper checks passed
  • CSharpier check passed for all six touched files

The repository-wide CSharpier check still reports pre-existing drift in unrelated files tracked separately by #57.

@dv00d00
dv00d00 merged commit cf6f488 into main Aug 20, 2026
1 check passed
@dv00d00
dv00d00 deleted the refactor/split-tokenizer-states branch August 20, 2026 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant