shibing624/similarity

What it solves

This project provides a Java-based toolkit for computing similarity scores between text strings across different granularities, from single words to full paragraphs. It also offers tools for sentiment analysis and finding approximate words using word embeddings.

How it works

The library implements a variety of algorithms categorized by text length:

• Words: Uses methods like Cilin encoding, Chinese semantic methods, and edit distance.
• Phrases: Calculates similarity based on shared characters and their positions.
• Sentences: Employs morpho-similarity (combining part-of-speech and word order) and various edit distance algorithms.
• Paragraphs: Utilizes cosine similarity (with TF-IDF and part-of-speech weighting), Jaccard, Euclidean, and SimHash with Hamming distance.
• Sentiment: Analyzes word polarity based on the HowNet semantic primitive tree.
• Approximate Words: Integrates Word2vec for vector-based synonym recommendation.

Who it’s for

Java developers who need to integrate text similarity measurement, sentiment analysis, or synonym discovery into their applications without implementing these NLP algorithms from scratch.

Highlights

• Multi-granularity support: Dedicated algorithms for words, phrases, sentences, and paragraphs.
• Low coupling: Internal modules are designed to be loosely coupled for easier maintenance.
• Lazy loading: Models are loaded only when needed to improve performance.
• Customizable: Supports training on custom corpora for Word2vec.
• Comprehensive distance metrics: Includes Jaro-Winkler, Manhattan, and Sørensen–Dice coefficients for paragraph comparison.