Slopo: Detecting Non-Exact Code Duplication with Embedding Models

Slopo uses embedding models to find non-exact code duplication

Slopo is a lightweight CLI tool designed to detect code duplication that is not an exact copy-paste. While traditional tools easily spot identical strings of code, Slopo uses embedding models to identify snippets that are written similarly but may exist far apart in a codebase, across different modules, or separated within large files.

Supported Languages

Slopo currently supports the following languages:

  • Python
  • TypeScript
  • JavaScript
  • Java
  • Kotlin
  • C#
  • Go
  • Rust

How the embedding-based detection works

Slopo identifies potential duplicates by calculating an embedding for every code unit and searching for pairs with high cosine similarity. This approach focuses on structural and semantic similarity rather than literal text matching.

The Analysis Pipeline

Detection occurs in two primary filtering passes:

  1. Similarity Threshold: Slopo first filters out code unit pairs whose embeddings are not similar enough based on a configurable similarity_threshold (cosine similarity ranging from -1 to 1).
  2. Reranking and Boosting: Similar pairs are grouped into clusters. These clusters are then reranked using a boost based on the distance between the code units in the codebase.
    • Cross-file distance: A boost (up to 15%) is applied based on the number of directory hops between files.
    • Same-file distance: A boost (up to 10%) is applied based on the distance in lines of code.

Clusters whose highest-scoring pair does not meet the rerank_threshold are filtered out.

Handling Exact Copies

While the tool focuses on non-exact duplication, it also detects exact copies. To keep reports clean, Slopo lists the code once and provides all paths where it appears, rather than repeating the identical snippet multiple times.

Integration into Development Workflows

Slopo is designed to be used as a precursor to AI-assisted refactoring. Because similar code is not always a duplicate that requires action, the tool provides a workflow for verifying and ignoring results.

Recommended Workflow

  1. Initial Analysis: Run slopo index, slopo embed, and slopo analyze to generate a report.
  2. AI Verification: Use an AI coding agent to review the generated clusters and determine if they are true duplicates.
  3. Filtering: Discarded clusters are added to slopo.ignore.txt via the AI agent to ensure they do not reappear in future reports.
  4. Refactoring: Use the verified clusters as a basis for refactoring, which can also be performed by an AI agent.
  5. Persistence: The ignore file can be committed to Git for cross-team consistency, while the local database (slopo.db) remains local.

Configuration and Technical Details

Slopo is installed via uv (uv tool install slopo) and requires an external embedding provider compatible with LiteLLM. The author recommends models dedicated to code, such as Voyage AI, for optimal results.

Key Configurable Parameters

  • source_dir: The directory containing the code to be indexed.
  • body_node_count_threshold: This parameter defines the minimum code complexity of a code unit based on the number of AST nodes in the body. This is used instead of simple text length to avoid reporting trivial, small code units.
  • similarity_threshold and rerank_threshold: Controls for the sensitivity of the duplication detection.
  • source_dir_exclude: .gitignore-style patterns to exclude specific directories (e.g., tests).

Community Insights and Perspectives

Users and developers in the Hacker News community highlighted several potential applications and limitations of the embedding-based approach:

"I can see this being useful before refactors, especially when the duplication is semantic rather than copy paste."

"I can imagine putting this into a pre push hook to keep things clean after an initial sweep."

Other contributors suggested exploring deterministic tiers before the embedding pass for efficiency, comparing the tool against simpler methods like BM25, or integrating the tool into code review processes to identify common constructs across a monorepo.

Sources

Related

  • Project
    rafal-qa/slopo

    A CLI tool that uses embedding models to detect non-exact code duplication across a codebase to help developers identify and refactor similar but not identical code.

  • Dispatch
    Large-scale Near-deduplication Behind BigCode

    Hugging Face details the use of MinHash and Locality Sensitive Hashing (LSH) to perform large-scale near-deduplication for the BigCode project, demonstrating that removing near-duplicate data improves model performance and training efficiency.

  • Dispatch
    Combating AI Slop: A Deterministic Approach to AI-Generated Code Smells

    Explore aislop, a CLI tool designed to detect and fix common antipatterns left by AI coding agents without relying on LLMs for the analysis.

  • Project
    arabold/docs-mcp-server

    Grounded Docs MCP Server is a local, version‑aware documentation indexer that scrapes official docs (web, GitHub, npm, PyPI, local files), converts them to searchable text, and optionally adds vector embeddings. It exposes a simple MCP (SSE) API so AI coding assistants (Claude, Cline, Copilot, Gemini‑CLI, etc.) can retrieve exact documentation passages, reducing hallucinations while keeping data private.

  • Dispatch
    Codestral Embed model release

    Mistral AI released Codestral Embed, a code‑specialized embedding model that outperforms existing code embedders on retrieval benchmarks and supports flexible dimensions and precisions.