MaartenGr/BERTopic

What it solves

BERTopic is designed to extract interpretable topics from a collection of documents. It solves the problem of traditional topic modeling by using dense clusters and a class-based TF-IDF (c-TF-IDF) procedure to ensure that topic descriptions remain meaningful and keep important words.

How it works

The project uses a modular pipeline consisting of several steps:

1. Embedding: Documents are converted into vectors using models like sentence-transformers.
2. Dimensionality Reduction: High-dimensional embeddings are reduced (typically using UMAP).
3. Clustering: Reduced embeddings are grouped into topics (typically using HDBSCAN).
4. Tokenization: Topics are tokenized to prepare for weighting.
5. Weighting: A c-TF-IDF procedure is used to weight tokens.
6. Representation: Topics are represented using the weighted tokens, with options to further refine these using LLMs (like GPT-4) or other representation models.

Who it’s for

It is intended for data scientists and researchers who need to analyze large volumes of text or multimodal data to discover underlying themes, trends, and patterns.

Highlights

• Highly Modular: Every step of the pipeline (embedding, reduction, clustering, etc.) can be swapped or removed.
• Versatile Modeling: Supports guided, supervised, semi-supervised, zero-shot, hierarchical, and dynamic topic modeling.
• LLM Integration: Can use LLMs to generate high-quality labels and summaries for the extracted topics.
• Multimodal Support: Capable of performing topic modeling on images as well as text.
• Rich Visualizations: Includes built-in tools to visualize topic hierarchies, document distributions, and topics over time.