Skip to content

Add automated checks for text cleanup and tokenization behavior #10

Description

@CocoHusky

Context

Rapid Reader depends on deterministic text cleanup, simplification, tokenization, punctuation pauses, and long-word rendering behavior. The README has a strong manual verification checklist, but these text-processing rules are good candidates for lightweight automated tests.

Proposed work

  • Add focused tests for cleanup rules such as code block replacement, inline code replacement, URL replacement, noisy symbol removal, link removal, hyphen splitting, and numeric citation stripping.
  • Add tests for token counts and edge cases like empty documents, long words, punctuation-heavy text, and Markdown tables.
  • Document the test command in the README if it differs from the existing build/check flow.

Acceptance criteria

  • Core cleanup/tokenization behavior can be verified without opening Obsidian.
  • Manual verification remains focused on UI behavior.
  • The tests do not require network access or a real vault.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions