Hugging Face Search Architecture for Papers with Code

Hugging Face has implemented a hybrid search system for Papers with Code to make AI research more accessible, combining keyword-based lexical search with vector-based semantic search. This architecture ensures that users can find research via exact titles or arXiv identifiers while also retrieving conceptually related work, even when specific keywords are absent from the paper.

Hybrid Search Architecture

Papers with Code utilizes a hybrid search system that combines the strengths of keyword search for exact mentions and vector search for semantic similarity. The system uses a PostgreSQL database for fast lexical baselines and pgvector for dense embeddings, combining the two using the Reciprocal Rank Fusion (RRF) algorithm.

The Retrieval Pipeline

For every query, the system executes two parallel branches:

  1. Lexical Branch: Retrieves up to 50 candidates using weighted PostgreSQL full-text search.
  2. Semantic Branch: Retrieves up to 50 candidates from pgvector using a cosine-distance search over an active generation of embeddings.

These results are merged using weighted RRF with equal branch weights and a rank constant of $k=60$. To maintain deterministic identity behavior, the system ensures exact titles and arXiv IDs remain at the top of results, while a method taxonomy handles navigational requests (e.g., "the original BERT paper").

Technical Stack and Infrastructure

To balance throughput for corpus building and low latency for live queries, Hugging Face split the search infrastructure into offline and online components.

Offline Corpus Build (Hugging Face Jobs)

Full-corpus embedding is treated as a batch workload. Hugging Face Jobs provide burstable GPU compute (specifically NVIDIA L4 GPUs) to process the paper corpus. The process involves:

  • Exporting paper data from a PostgreSQL snapshot into JSONL shards.
  • Loading a pinned model revision of Qwen/Qwen3-Embedding-0.6B.
  • Encoding documents in batches, sorting texts by length to reduce padding.
  • Truncating vectors to 256 dimensions using Matryoshka Representation Learning (MRL) and applying L2 normalization.

Durable Storage (Hugging Face Storage Buckets)

Storage Buckets act as the connective tissue between the database, ephemeral Jobs, and the production index. By organizing artifacts under immutable run prefixes with manifests and checksums, the system achieves:

  • Reproducibility: Tracing generations back to specific snapshots and model revisions.
  • Safe Retries: Resuming work from completed shards.
  • Controlled Rollout: Validating coverage and building HNSW indices before atomically activating a new generation.

Online Search (Hugging Face Inference Endpoints)

Live queries are embedded using an authenticated Inference Endpoint backed by Text Embeddings Inference (TEI). This endpoint uses the query prompt of the Qwen3 model to generate a normalized 256-dimensional vector.

To handle the "scale-to-zero" nature of the endpoint and avoid latency spikes during cold starts, the system implements a strict client-side policy: a one-second production timeout and a circuit breaker. If the semantic endpoint is unavailable or times out, the system immediately falls back to lexical search results.

Embedding Contract and Model Selection

To prevent subtle failures in embedding pipelines, Hugging Face treats the embedding format as a versioned API. Every paper is encoded as normalized title + "\n\n" + normalized abstract.

Model Specifications:

  • Model: Qwen/Qwen3-Embedding-0.6B (pinned to an exact revision).
  • Dimensions: 256 (selected via MRL to balance speed and storage).
  • Normalization: L2-normalized vectors.
  • Prompts: Distinct document prompts for corpus embedding and query prompts for live searches.

Operational Insights and Lessons Learned

Throughput vs. Latency

Separating throughput-oriented work (Jobs) from latency-sensitive work (Inference Endpoints) allows the system to optimize for cost and availability independently.

Vector Dimensionality

Using Matryoshka embeddings allowed the team to reduce dimensionality to 256. In pilot tests on 5,000 papers, this configuration achieved 0.9955 Recall@20 against exact search while using only 27% of the storage required for 1024-dimensional vectors.

Incremental Updates

While Jobs handle full rebuilds, an hourly incremental process uses the same Inference Endpoint (with the document prompt) to embed new or modified papers in batches of up to 500, keeping the index current without the overhead of a full GPU Job.

Related Papers Feature

Because document embeddings are already stored in PostgreSQL, the "Related Papers" feature on individual paper pages is implemented as a simple nearest-neighbor query, requiring no real-time model inference.

Sources

Related