Ternlight: Browser-Based Semantic Search with 7 MB Embedding Model

Ternlight enables high-performance semantic search directly in the web browser by utilizing a highly compressed embedding model that runs on the CPU via WebAssembly (WASM). By eliminating the need for server-side API calls, it allows for "search-as-you-type" functionality with millisecond latency and enhanced user privacy.

Technical Implementation and Architecture

Ternlight is not a Large Language Model (LLM), but a sentence encoder designed to transform text into 384-dimensional vectors. These vectors allow the system to determine the semantic relationship between two pieces of text using cosine similarity, enabling matches based on meaning rather than keyword overlap (e.g., matching "reset my password" with "I forgot my password").

Key technical specifications include:

• Model Origin: Distilled from MiniLM using ternary quantization-aware training.
• Inference Engine: Written from scratch in Rust and compiled to WASM SIMD for optimal CPU performance.
• Hardware Requirements: CPU-only; no GPU is required.
• Performance: Embeddings are generated in approximately 2.5ms to 5ms.

Model Tiers and Deployment

Ternlight is distributed as an npm package, removing the need for separate model download steps or backend infrastructure. It offers two size tiers to balance capability and performance:

Tier	Package	Size	Inference Speed
Base	@ternlight/base	7 MB	~5 ms
Mini	@ternlight/mini	5 MB	~2.5 ms

Developers can implement semantic search with minimal code:

import { embed, similar } from '@ternlight/base';

similar('easy weeknight dinner ideas', recipes, { topK: 3 });
// Returns ranked matches in ~5 ms with zero network latency

Use Cases and Practical Applications

The ability to run embeddings on-device opens several specific technical opportunities:

• Semantic Search: Implementing fast, local search for documentation (as seen in the React docs demo) or product catalogs.
• - FAQ and Intent Matching: Mapping user queries to predefined answers without sending data to a server.
• Clustering: Grouping similar texts locally on the client side.
• Privacy-Preserving Search: Ensuring user queries never leave the local device, which is critical for sensitive data.

Community Insights and Considerations

Community discussion highlights both the potential for distributed search and the concerns regarding client-side resource usage.

Integration Potential

Users suggested integrating Ternlight with other local-first technologies to create a fully static vector search ecosystem. One user proposed combining it with an Astro plugin to automatically parse HTML files and generate embeddings, while another suggested pairing it with portable HNSW (Hierarchical Navigable Small World) search via HTTP range queries across Parquet files to create a distributed search system not controlled by centralized corporations.

Resource and Privacy Concerns

While the privacy benefits are clear, some users expressed concerns about the impact on system resources:

"it's a bit startling to hear my fans go crazy when opening a webpage."

Additionally, there are concerns regarding the security implications of automatically downloading models into the browser, with some users noting that this could potentially be used to hog memory or distribute malicious code if not handled carefully.