fix: answer questions whose answers exist in the knowledge base

[HardeepAsrani]

Summary

Closes Codeinwp/hyve#298

The chatbot refused questions whose answers were sitting in the knowledge base. Two things caused it: ingestion was destroying the content's structure, and the similarity threshold was doing a job it cannot do.

Ingestion now preserves structure. Stripping tags used to fuse table cells ("Laundry$32.00/hr") and glue headings to paragraphs, and any long block without sentence punctuation (like a table) was silently dropped from the index entirely. Content is now extracted as structured text: table rows become "Laundry | $32.00/hr" lines, headings keep their own line, entities are decoded, and oversized blocks are split instead of thrown away. Hyve Connect uploads ship the same clean text.

Retrieval now lets the model judge. Measurements showed an absolute similarity score cannot separate answerable from unanswerable questions: a legitimate question can score lower against its correct page than an off-topic one scores against a tangential page. So the default threshold drops from 0.4 to 0.25 and acts as a noise floor, a new relative cutoff keeps only chunks scoring close to the query's best match, and the model (which sees the actual text) decides whether it can answer. Sites that saved a custom threshold keep their value.

Also fixed along the way:

• Context sent to the model is now ordered best match first (it was database order).
• Follow-up retrieval blends only the visitor's messages, so the bot's own replies (especially "Sorry, I can't help" fallbacks) no longer poison the search.
• New filters for tuning: hyve_context_score_ratio, hyve_chat_context_token_limit, hyve_chunk_token_size.

Verified on a live seeded site: "How much does laundry cost per hour?" went from refused to answered ("Laundry costs $32.00 per hour."), while off-topic questions are still correctly declined.

QA instructions

1. On a self-hosted site, create a page containing a pricing table (service names and rates) plus some regular paragraphs, and index it in the knowledge base.
2. Ask the chatbot about a specific rate from the table (e.g. "How much does laundry cost per hour?"). It should answer with the correct value. Before this fix it said it couldn't help.
3. Ask something clearly not in your content (e.g. "Do you sell wheelchairs?"). It should still decline.
4. Ask a question, then follow up with a topic-less one ("How much does it cost?"). The bot should resolve it against the earlier topic.
5. Re-index an existing page (edit and update it) and repeat step 2 on that content.
6. Check Settings → AI: a previously saved similarity threshold should be unchanged; on a fresh install the default is 0.25.
7. If the site uses Hyve Connect, edit and re-sync a page containing a table, then ask about a value from it.

[github-actions]

Plugin build for 0f49011 is ready 🛎️!

• Download Plugin - Download