feat(backend): route uncertain conversations to their category folder (#4043)

[ZachL111]

Summary

Improves AI folder assignment accuracy (#4043). When the model is unsure which folder a conversation belongs in, it now falls back to the folder that owns the conversation's category instead of the catch-all default, so an uncertain "finance" conversation lands in Work rather than the default folder.

What was missing

The issue notes folder assignment "isn't that accurate" and "does not seem to take into consideration the title/summary." Two things were going on:

• The conversation title and overview are already passed to the assignment prompt today, so that part is handled.
• The codebase already had an authoritative CATEGORY_TO_FOLDER_MAPPING in database/folders.py that folds every one of the ~30 conversation categories onto one of the three system folders (Work, Personal, Social). It was defined but used nowhere. The AI assignment relied entirely on fuzzy LLM matching against folder descriptions and ignored this mapping, so a conversation the model was unsure about (confidence below the 0.7 threshold) got dropped into the default folder even when its category clearly pointed at a system folder.

Fix

• Added database.folders.resolve_category_folder_id(category, user_folders): uses the existing CATEGORY_TO_FOLDER_MAPPING to return the user's system folder for a conversation's category, or None when the category is unknown or the user has no folder for that bucket.
• build_folders_context now surfaces each folder's category with a "[home for X conversations]" annotation, and the prompt tells the model to prefer the category-aligned folder when the conversation's category matches.
• validate_folder_assignment takes the category-aligned folder and uses it as the preferred fallback over the default when the model returns an invalid folder or is below the confidence threshold. When there is no category-aligned folder the behavior is unchanged.
• process_conversation passes the resolved category folder through. The change is additive: confident valid assignments and custom-folder matches are untouched, and the new parameter is optional on every function.

Testing

• New tests/unit/test_folder_assignment_category.py (14 tests): the category resolver (exact and folded categories, case-insensitivity, unknown categories, missing system folder), the category annotation in build_folders_context, the category-preferred fallback in validate_folder_assignment for both the low-confidence and invalid-folder paths, and assign_conversation_to_folder threading the category folder through end to end while leaving confident assignments untouched. Registered in test.sh.
• Proven red against the pre-change version (13 of 14 fail without the change), green after.
• tests/unit/test_folder_name_enrichment.py and tests/unit/test_folder_conversations_malformed.py still pass (20 tests), confirming no regression.

Refs #4043.

[cubic-dev-ai]

No issues found across 5 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}

[Git-on-my-level]

Thanks for the thoughtful patch — this looks like a sensible direction for #4043.

Maintainer review notes:

• The implementation reuses the existing CATEGORY_TO_FOLDER_MAPPING instead of inventing a new taxonomy, which keeps the behavior aligned with the current folder model.
• I like that uncertain/invalid LLM folder picks now fall back to the category-aligned system folder, while confident picks can still select a better matching custom folder.
• The added unit coverage exercises the resolver, prompt context, validation fallback behavior, and the end-to-end assignment path.
• CI is currently green.

I’m leaving this as a positive signal rather than a formal approval because it changes backend conversation-to-folder routing / user data organization behavior, and the product choice (category-aligned system folder vs. default folder when uncertain) should get maintainer confirmation before merge.

One small non-blocking cleanup to consider: the prompt still says “When in doubt, use the DEFAULT folder,” while the validator now deliberately maps low-confidence responses to the category-aligned folder when available. The code path is covered by tests, but updating that wording could make future prompt behavior easier to reason about.

[ZachL111]

Thanks David, good call. I updated the instruction in fcb0cb5: "When in doubt, use the DEFAULT folder" is now "When no folder clearly matches, prefer the folder for the conversation's category if one is listed above, otherwise use the DEFAULT folder", so the prompt and the validator fallback now describe the same behavior.

The product choice you flagged (category-aligned system folder vs the default folder when the model is uncertain) is yours to confirm. If the team would rather keep the old default-only fallback, that is a one line change to the validator and I am happy to make it.

[kodjima33]

Backend feature: route uncertain conversations to category folder — approve only.

[Git-on-my-level]

Thanks for the update — the main instruction now matches the category-aligned fallback much better.

One remaining wording nit before a maintainer makes the product call: the confidence guidance still says “below 0.7 means use DEFAULT”, while the validator now intentionally routes low-confidence/invalid picks to the category-aligned folder when one exists. Could you update that sentence too so future prompt behavior and the safety fallback are fully consistent?

I still see this as a positive implementation direction with good unit coverage and green checks, but I’m not formally approving because this changes backend conversation-to-folder routing / user data organization behavior and should remain with human maintainer review.

[ZachL111]

Done in fb01ab2. Updated the confidence guidance from "below 0.7 means use DEFAULT" to "below 0.7 routes to the conversation's category folder if one is listed, otherwise the DEFAULT folder", so the prompt and the validator fallback are now fully consistent.

As before, the product call on category-aligned vs default when the model is uncertain is yours; happy to revert the fallback to default-only if the team prefers.

[Git-on-my-level]

Thanks for the quick follow-up — the remaining prompt wording is now aligned with the validator behavior.

Automated maintainer review notes for this head:

• The change reuses the existing CATEGORY_TO_FOLDER_MAPPING rather than adding a parallel taxonomy.
• Low-confidence/invalid folder selections now fall back to the category-aligned system folder when available, while confident valid custom-folder choices are still preserved.
• The added unit tests cover the resolver, folder-context annotation, validator fallback paths, and end-to-end argument threading.
• Checks are green, and I don’t see a security/supply-chain concern in the diff.

No further code-review blocker from me. I’m still not giving formal automated approval because this changes backend conversation-to-folder routing / user data organization behavior and is under the repo’s human product/security review gate, but this looks ready for a maintainer’s final decision.