Tell a construction from a keyword when cleaning up after the fact - #214
Merged
Merged
Conversation
…n article The anchored gate shipped and the hourly cron wrote its first 162 keywords across 11 sites. The vendor and junk classes are gone. Reading the actual output found two things the fixtures could not have. **Partial matches leaked.** Eight keywords, all one shape: one generic word of a multi-word subject matched, and a word from the generic commercial vocabulary rescued it. "open standards" matched on "open", "software" did the rest, and logicsrc was queued to write about OBS Studio. "supply chain security" matched on "supply" and admitted "industrial automation supply". "online shopping" matched on "online" and admitted "quickbooks online pricing". So a partial match on a multi-word subject may now only be rescued by an anchor the site itself supplied — never by the defaults, which are generic by construction. A complete match is unaffected, including a single-word subject, where "iptv alternatives" is a real query. **And the cross-product floor was publishing constructions.** That one is mine and it was new. Used as per-subject filler it produced "saving money pricing", "deals platform", "coordination d0rz" and "ai content loop" — on-niche, gate-passing, and not topics anybody searches. bl0ggers' niche is "human-in-the-loop AI publishing", so its derived modifiers are literally "human" and "loop", and crossing a subject with those yields nonsense. The crosses' real value was always upstream: as DataForSEO seeds they are what turns "peptide" into "peptide merchant account", and they still seed every expansion. They are simply no longer published on the strength of being grammatically adjacent to the niche. They remain the floor, but site-level rather than per-subject — only a run that would otherwise insert nothing falls back to them, which keeps "a blog with every upstream down still publishes" without letting constructions pad a healthy run. The result reports usedCrossFloor so a site sitting on that for weeks is visible. All eight leaked keywords are now regression cases, with the on-niche keywords from the same run that must still pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013cbcXdpLFUgJZuVfCB4Gzv
Companion to purge-offniche-keywords.ts, for rows written before the gate was tightened. Two verdicts, both computed with the real planner rather than approximated in SQL. The second one needed care. A construction cannot be found by checking for a null search volume, which was the obvious approach: the buyer-journey model also returns keywords with no volume, and those are the best output in the run — "zeek vs suricata for network security monitoring" and "langchain alternatives for agents" both sit in the same null bucket as "deals platform". So a construction is identified by fingerprint against the cross set the planner would build for that site, which is exact rather than heuristic. Ran against the 162 rows from the first production cron: kept 53, removed 95 constructions and 14 partial-match leaks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013cbcXdpLFUgJZuVfCB4Gzv
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| console.log("\n-- SQL:"); | ||
| for (let i = 0; i < allIds.length; i += 200) { | ||
| console.log( | ||
| `delete from lx_keyword where status='queued' and id in (${allIds |
ThreatCrush Security Scan40 finding(s) HIGH/CRITICAL: 3 | MEDIUM: 28 | LOW: 9
Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
scripts/purge-constructed-keywords.ts, the tool used to clean the 162 rows the first production cron wrote before #213 tightened the planner.A construction cannot be found by checking for a null search volume — the buyer-journey model also returns keywords with no volume, and those are the best output in the run.
zeek vs suricata for network security monitoringsits in the same null bucket asdeals platform. So constructions are identified by fingerprint against the cross set the planner would build for that site, which is exact.Ran against production: kept 53, removed 95 constructions and 14 partial-match leaks.
🤖 Generated with Claude Code
https://claude.ai/code/session_013cbcXdpLFUgJZuVfCB4Gzv