clean_for_speech - post process markdown text fit for reading aloud - #4
clean_for_speech - post process markdown text fit for reading aloud#4factorial wants to merge 1 commit into
Conversation
| // Matches RFC 3986 scheme followed by `://` and a run of non-whitespace, | ||
| // non-quote/angle-bracket characters. | ||
| static URL_PATTERN: LazyLock<Regex> = | ||
| LazyLock::new(|| Regex::new(r#"\b[a-zA-Z][a-zA-Z0-9+.-]*://[^\s<>"]+"#).expect("valid regex")); |
There was a problem hiding this comment.
Since we're not validating URLs, I think any schema followed by :// followed by anything up until whitespace can be considered a URL
| let trimmed = matched.trim_end_matches(TRAILING_PUNCTUATION); | ||
| let trailing = &matched[trimmed.len()..]; | ||
|
|
||
| match reqwest::Url::parse(trimmed).ok() { |
There was a problem hiding this comment.
This is very LLM. I have no idea why you would do this
There was a problem hiding this comment.
Ah, this is leftover from when originally I was leaving the domain behind of any URL returned. OpenAI's web search is very fond of returning exact source URLs. For a minute I thought it'd be nice to say "example.com" but ultimately decided removing them entirely was better. Will clean up.
|
|
||
| // Characters that are almost always sentence punctuation that may come | ||
| // after a URL (e.g. "(see https://example.com)."). | ||
| const TRAILING_PUNCTUATION: &[char] = &['.', ',', ';', ':', '!', '?', ')', ']', '\'', '"']; |
There was a problem hiding this comment.
I like this idea, but I think it's done incorrectly. I would grab the entire URL. If it ends in any of the above characters (probably not \' lol), remove all BUT that last character. We keep the trailing punctuation so it's spoken correctly.
| } | ||
|
|
||
|
|
||
| /// Removes URLs from `text`. Useful for preventing LLM responses being read aloud |
There was a problem hiding this comment.
This was actually mine lol but if you prefer its removal that's OK
| let mut list_item_seen: Vec<bool> = Vec::new(); | ||
| let mut skip = Skip::None; | ||
|
|
||
| let push_block_separator = |output: &mut String| { |
There was a problem hiding this comment.
This is overcomplicated LLM garbage. You want to add a sentence terminator (.) after very markdown block; no blocks exist mid-sentence
| if is_first { | ||
| push_block_separator(&mut output); | ||
| } else { | ||
| output.push_str("; "); |
There was a problem hiding this comment.
Why not a comma? Should be a const
|
|
||
| let mut output = String::new(); | ||
| let mut list_item_seen: Vec<bool> = Vec::new(); | ||
| let mut skip = Skip::None; |
| @@ -0,0 +1,293 @@ | |||
| use pulldown_cmark::{Event, Options, Parser, Tag, TagEnd}; | |||
|
|
|||
| enum Skip { | |||
There was a problem hiding this comment.
NestedContext. None is not necessary
| match event { | ||
| Event::Start(tag) => match tag { | ||
| Tag::CodeBlock(_) => { | ||
| skip = Skip::CodeBlock(String::new()); |
There was a problem hiding this comment.
You enter the code block. You push to the context stack. You do nothing else until you see the codeblock end token. At that point, you pop the stack, and IFF the stack is empty, you write the codeblock substitution string.
Same for all other blocks
2 participants
Since LLM responses (especially OpenAI) often include markdown that sounds bad when read aloud, post processing function for cleaning up the response for speech.