Skip to content

Creating Topic Taxonomies #2526

Description

@TheItCrOw

Feature request

Hey @MaartenGr,

first off: great repository! It was really easy to get into it and we're using BERTopic extensively, which brings me to my feature request (that I will follow up with a PR -> Update: It's here :):

create_topic_taxonomy(): controllable multi-level topic hierarchies

We've been building topic taxonomies for our use case and ran into a gap between the existing options:

  • hierarchical_topics() gives a full dendrogram but no control over the number of levels
  • reduce_topics() can reduce to a target count but isn't really designed for building hierarchies - calling it repeatedly works but produces a lot of single-child branches and the quality of the hiearchy is questionable.

What we needed was a way to say "give me 3 levels with roughly X, Y, Z topics each, and make sure every parent has at least N children."

Motivation

We're using BERTopic a lot, also in production, to analyze textual data, but we need levels and a topic hierarchy, which is why I'd like to contribute directly to the repo.

Your contribution

So we built create_topic_taxonomy(), which:

  • Takes nr_topics_per_level (e.g. [20, 5]) to define the shape
  • Enforces a min_children constraint per parent (default 2) to avoid degenerate branches
  • Optionally blends document centroids with topic embeddings (configurable weight) for richer topic vectors when building the hierarchy, which works really well for us.
  • Clusters bottom-up using agglomerative clustering + a repair step for undersized clusters
  • Labels parents via c-TF-IDF on merged documents
  • Returns a clean DataFrame with Topic_ID, Level, Parent_ID, etc.
  • Does not touch the fitted model - it's a read-only lookup table on top of the leaves

The method sits right next to hierarchical_topics() and follows existing code patterns. PR incoming with tests and docs.


We've added a dedicated documentation page to the mkdocs (Topic Taxonomy), the actual implementation is only a single method. :)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions