MaartenGr/KeyBERT

KeyBERT – Keyword extraction with BERT embeddings

What it is – A tiny Python library that turns any text document into a list of the most representative words or phrases. It does this by:

Getting a document embedding from a BERT‑style model (e.g., a Sentence‑Transformer).
Getting embeddings for every candidate n‑gram in the text.
Ranking candidates by cosine similarity to the document embedding.

Because the heavy lifting is done by pre‑trained language models, you only need a few lines of code and no model‑training.

Core features

Feature	What it does
Zero‑training keyword extraction	Uses off‑the‑shelf BERT embeddings; just `pip install keybert` and call `extract_keywords`.
Customizable n‑gram length	`keyphrase_ngram_range` lets you ask for single words, 2‑grams, 3‑grams, etc.
Diversity controls	`use_maxsum` (max‑sum distance) and `use_mmr` (maximal marginal relevance) let you trade off relevance vs. variety.
Multiple embedding back‑ends	Supports Sentence‑Transformers, Flair, spaCy, Gensim, TensorFlow‑Hub’s Universal Sentence Encoder, or a lightweight Model2Vec‑based setup.
LLM‑based extraction (KeyLLM)	Optional wrapper that sends the text to an OpenAI chat model and returns its keyword suggestions.
Highlighting	Quick visualisation of extracted keywords inside the original document.

Quick start

pip install keybert               # core package
pip install keybert[flair]        # optional back‑ends (flair, spacy, gensim, use)

from keybert import KeyBERT

doc = "Supervised learning is the machine learning task of learning a function …"
kw_model = KeyBERT()                     # defaults to a small English Sentence‑Transformer
keywords = kw_model.extract_keywords(doc)
print(keywords)  # [('learning', 0.46), ('algorithm', 0.45), …]

Change n‑gram size: kw_model.extract_keywords(doc, keyphrase_ngram_range=(1,2)) Add diversity: kw_model.extract_keywords(doc, use_mmr=True, diversity=0.7) Highlight: kw_model.extract_keywords(doc, highlight=True)

When to use it

Document summarisation – pull out the most salient terms for reports, articles, or research papers.
Search‑engine indexing – generate tags for fast lookup.
Pre‑processing for downstream NLP – feed keywords into clustering, recommendation, or topic‑model pipelines.
Rapid prototyping – no need to train a custom keyphrase model; works out‑of‑the‑box with a single line.

Limitations

Quality depends on the embedding model – a poor‑performing transformer will give noisy keywords.
Max‑sum / MMR are combinatorial – nr_candidates must stay small for reasonable runtimes.
LLM mode requires an external API key and incurs cost.
No built‑in multilingual tokenisation – you need a multilingual embedding model (e.g., paraphrase-multilingual-MiniLM-L12-v2).