Ternlight: 7 MB Browser-Based Embedding Model for Local Semantic Search

Ternlight is a high-performance, client-side embedding model designed to bring semantic search to the web browser without requiring an API or server-side infrastructure. By leveraging WebAssembly (WASM) and ternary quantization-aware training, the project provides a compact footprint (5-7 MB) and near-instant inference times (~2.5ms to 5ms per embedding).

Technical Architecture and Implementation

Ternlight is not a Large Language Model (LLM), but a sentence encoder that transforms text into a 384-dimensional vector. This allows developers to calculate cosine similarity between vectors to determine the semantic relationship between two pieces of text, regardless of whether they share common keywords.

Core Engineering

  • Distillation and Quantization: The model was distilled from MiniLM using ternary quantization-aware training to minimize size while maintaining utility.
  • Inference Engine: The inference engine was written from scratch in Rust and compiled to WASM with SIMD (Single Instruction, Multiple Data) support to maximize CPU performance.
  • Deployment: The project is distributed as a single npm package, eliminating the need for separate model download steps or external server calls.

Model Variants

There are two tiers available via npm:

  • @ternlight/base: A 7 MB model providing more capable embeddings with an average inference time of ~5 ms.
  • @ternlight/mini: A 5 MB model optimized for size, with an average inference time of ~2.5 ms. This variant uses 256-element vectors internally and projects them up to 384 for compatibility.

Integration and Usage

Developers can implement semantic search in three lines of code using the @ternlight/base package. The library provides embed and similar functions to enable ranked matches based on semantic similarity.

import { embed, similar } from '@ternlight/base';

similar('easy weeknight dinner ideas', recipes, { topK: 3 });
// → ranked matches · ~5 ms · zero network

Use Cases and Community Insights

The community has identified several high-value applications for on-device embeddings, ranging from privacy-focused search to hybrid retrieval systems.

Local Semantic Search and Privacy

Local execution ensures that user queries remain on the device, providing inherent privacy and reducing latency. This is particularly useful for fast, "search-as-you-type" experiences in product bases or documentation sites.

Hybrid Retrieval Systems

Technical discussions suggest combining Ternlight with other browser-based tools to create sophisticated search architectures. For example, combining Ternlight's semantic search with SQLite's FTS5/BM25 (Full Text Search) via tools like absurder-sql allows for Reciprocal Rank Fusion (RRF), enabling hybrid retrieval that combines keyword and semantic matching entirely on the client side.

Distributed Search Ecosystems

Some users suggest integrating Ternlight with portable HNSW (Hierarchical Navigable Small World) search and Parquet files to create a distributed search ecosystem that is not dependent on centralized corporate APIs.

Performance Considerations

While the highly optimized WASM engine provides fast inference, some users have noted that the initial loading of the model can be CPU-intensive, causing system fans to trigger on some devices. Additionally, while the mini variant is highly efficient, some users have reported that the accuracy of similarity search may vary depending on the complex nature of the query.

Sources

Related

  • Dispatch
    ChonkLM: Bringing Tiny Language Models to the Browser via WebGPU

    An exploration of ChonkLM, a browser-based inference runtime that allows users to run small language models entirely offline using WebGPU.

  • Dispatch
    EmbeddingGemma 300M release notes

    Google released EmbeddingGemma, a 308M‑parameter multilingual embedding model that tops the MTEB benchmark for sub‑500M models and is optimized for on‑device use.

  • Dispatch
    Building a High-Performance LLM Inference Engine from Scratch with tiny-vLLM

    An exploration of tiny-vLLM, a C++ and CUDA-based educational project that teaches the internals of LLM inference, from Safetensors loading to PagedAttention.

  • Dispatch
    Semble: Optimizing Code Search for AI Agents with 98% Token Reduction

    Explore how Semble replaces inefficient grep-and-read workflows with a fast, CPU-based semantic search library designed specifically for coding agents.

  • Project
    jmaczan/tiny-vllm

    tiny‑vllm is a C++/CUDA LLM inference engine (a smaller, educational version of vLLM) that loads a Llama 3.2 1B Instruct model from safetensors, runs the full forward pass on GPU (prefill + decode), implements KV‑cache, static/continuous batching, FlashAttention‑style online softmax and PagedAttention, and includes a step‑by‑step tutorial for learning or teaching GPU‑based LLM serving.