enjalot/latent-scope

What it solves

Latent Scope provides a streamlined workflow for visualizing and exploring unstructured text and image datasets. It solves the problem of manually analyzing large volumes of data by automating the process of turning high-dimensional embeddings into an interactive, labeled 2D map that users can explore to find patterns and insights.

How it works

The tool follows a multi-step pipeline that can be managed via a web interface or a command-line interface:

1. Embedding: Unstructured data is converted into vectors using local models (HuggingFace) or APIs (OpenAI, Cohere, etc.). It supports dense embeddings and ColBERT late-interaction embeddings.
2. Projection: High-dimensional vectors are reduced to 2D coordinates using UMAP.
3. Clustering: Points are grouped using methods like HDBSCAN, KMeans, Gaussian Mixture, or EVoC.
4. Labeling: An LLM is used to summarize the clusters and assign human-readable labels to them.
5. Exploration: The resulting "scope" is served via a web UI where users can interactively browse the annotated data, perform similarity searches, and compare different projection/clustering configurations.

Who it’s for

It is designed for AI researchers, data scientists, and analysts who need to explore unstructured datasets to understand their composition, identify themes, or curate data for machine learning tasks.

Highlights

• Multimodal Support: Works with both text and image datasets, including a specialized image map for CLIP-style embeddings.
• GPU Acceleration: Optional NVIDIA GPU support for UMAP and clustering via cuML to handle larger datasets.
• LLM-Powered Labeling: Automatically generates cluster descriptions using chat models.
• Flat-File Architecture: Stores all intermediate data and metadata in standard flat files (Parquet, JSON, H5) for easy portability and reproducibility.
• Agent-Friendly: Includes a cross-provider agent skill allowing AI coding agents (like Claude Code or Cursor) to drive the data mapping pipeline.