Skip to content

approximate_distribution: add target-based min_similarity ("at most N unassigned documents") #2502

Description

@pidefrem

Feature request

approximate_distribution() requires manually specifying min_similarity. Like the threshold parameter in reduce_outliers, users must iterate to find a value that produces an acceptable number of documents with zero topic distribution.

Add an outliers_nb_target parameter that auto-searches for the optimal min_similarity:

# Current: manual trial-and-error
distributions = topic_model.approximate_distribution(docs, min_similarity=0.1)
# Count zeros... too many. Try 0.05... too few. Try 0.08...

# Proposed: specify the target directly
distributions = topic_model.approximate_distribution(docs, outliers_nb_target=50)

Motivation

Users know their target — "I want at most 50 documents without any topic assignment" — not the internal min_similarity value that achieves it. The same trial-and-error UX problem as reduce_outliers (see related issue).

Your contribution

I can submit a PR that adds:

  • outliers_nb_target (int): target number of zero-distribution documents
  • Performance optimization: refactor the internals so that the expensive similarity matrix is computed once and reused across all binary search iterations

Cannot set both min_similarity and outliers_nb_target. Backward compatible: defaults to None.

I've already been prototyping this in my fork, so I can open a PR quickly if this looks like a good direction.


Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions