remove_words enhancements

[ebhills]

Current behavior of remove_words function needs changes:

Requirements:

• tokenize_to_remove should default to true. change in the code as well as the schema doc string. Also more than one space should be reduced to a single space so that the tokens do not have spaces in them. If the inputs are already lists, then the tokenize step should be ignored
• Add new parameter: characters_to_consider: all | letters_numbers_only (default). Default behavior is to remove anything [^a-z0-9] from tokens so that non alphanum characters are not included in the tokens and thus matches occur even when intial strings have miscellaneous punction, /s etc.
• Regardless of characters_to_consider, any punctuation or spaces at the beginning or end of the input and to_remove values should be removed before tokenization.
• If either of input or to_remove are lists and the other is just a str, should convert/tokenize the str and run the remove in the list mode.
• Add defults for ignore_case at True, and note this in the schema
• Add tests for all combinations. one example:
  input: Any, Words, Overlap?
  to_remove: Any Words Overlap
  output: '' (empty string)

[ebhills]

text_clean = text.translate(str.maketrans('', '', string.punctuation))

This single line of code is the most efficient, built-in way to remove all punctuation from a string in Python.

Here is exactly how it works, broken down from the inside out:

1. string.punctuation
   The string library has a built-in constant called punctuation. It is literally just a pre-defined string containing all common punctuation marks.
   If you were to print(string.punctuation), you would see:
   !"#$%&'()*+,-./:;<=>?@[]^_{|}~`

2. str.maketrans('', '', string.punctuation)
   maketrans creates a "translation table" (essentially a dictionary of instructions) that tells Python how to replace or remove characters. It accepts up to three arguments:

Argument 1 (Replace these): '' (We leave this empty because we aren't replacing anything).

Argument 2 (With these): '' (Also empty).

Argument 3 (Delete these): string.punctuation (This is the magic part).

By passing the punctuation string as the third argument, you are generating a translation table that simply says: "If you see any of these characters, delete them."

3. text.translate(...)
   Finally, the .translate() method applies that instruction table to your actual text. It scans through your string character-by-character; if a character isn't in the deletion table, it keeps it. If it is (like a comma or hyphen), it drops it.

[ebhills]

@mborodii-prog - I want to investigate using this as a global technique wherever we compare / replace text.

[mborodii-prog]

@ebhills # Key Behavioral Differences

The two approaches are not equivalent:

Behavior	re.sub(r'[^a-zA-Z0-9]', '', s)	translate(maketrans('', '', string.punctuation))
Removes spaces	Yes	No
Removes Unicode punctuation (—, ", ')	Yes	No (ASCII only)
Removes digits	No	No
Scope	Everything non-alphanumeric	Only string.punctuation chars

The current regex is stricter — it also removes spaces (fine for single tokens) and Unicode punctuation. The translate approach would silently miss em-dashes, curly quotes, and other non-ASCII punctuation.

Where It Should NOT Be Applied

Location	Reason
_strip_boundary_punc (extract.py:620)	Only strips leading/trailing punctuation — translate would strip all of it
clean_whitespaces (recipe_wrangles/main.py:392)	Handles Unicode whitespace, not punctuation
replace (recipe_wrangles/main.py:1610)	User-defined find/replace — unrelated
compare.py overlap/contrast	.lower() only; adding punctuation removal would change comparison semantics in a breaking way
Variable name sanitization (create.py:373, main.py:1137)	Replaces with _, not deletion — different intent entirely

---

Recommendation

The translate technique is faster than regex for purely ASCII punctuation removal, but it is a narrower tool than what most of these sites currently use. Applying it globally would introduce subtle regressions on Unicode input.

Better alternative: str.isalnum()

For _normalize_token specifically, the strongest candidate is:

def _normalize_token(token):
    token = ''.join(c for c in token if c.isalnum())
    if ignore_case:
        token = token.lower()
    return token

Why isalnum() wins here:

• Faster than regex
• Handles Unicode correctly (no silent misses on —, ", etc.)
• Replaces both space-removal and punctuation-removal in one pass
• No import required