Exclude scripts, styles and chrome from word_count

[amedipiran]

seo_extractor.py counts words with soup.get_text() on the whole document. That includes the contents of <script> and <style>, plus the navigation, header and footer that repeat on every page.

On a WooCommerce site with a large mega menu I measured 1333 words on a product page, of which 774 were chrome and inline JS. The homepage went from 1250 to 461.

This quietly breaks two things that depend on word_count:

• issue_detector flags low content below 300 words. On a site with a decent menu that threshold can never be reached, so the check silently never fires.
• word_count carries 0.10 weight in near-duplicate detection, computed on a number that is mostly boilerplate.

The fix strips script, style, noscript, template, nav, header, footer and aside, then prefers <main> over <body>. It works on a copy.copy of the soup, so the caller keeps using the same soup afterwards for links, headings and images. I checked that: after counting, the link count on the original soup is unchanged.

Worth noting this will lower word_count across the board, so any thresholds people have tuned against the old numbers will behave differently. That seemed better than leaving the number meaningless, but your call.

Generated with Claude Code