Building LLM-Powered Web Apps with Client-Side Technology

Local LLM Integration for Web Applications

Building web applications using exclusively local models and client-side technologies eliminates the need for expensive API calls to providers like OpenAI or Anthropic. This approach provides three primary advantages: zero additional compute costs for the developer, enhanced user privacy as data never leaves the local machine, and the potential for increased speed by removing HTTP call overhead, although this may be offset by the limitations of the user's hardware.

Implementing a Local RAG Pipeline

Retrieval-Augmented Generation (RAG) allows users to "chat with their documents" by extracting information from unstructured data. A fully local RAG pipeline consists of two main phases: data ingestion and retrieval/generation.

Data Ingestion

Data ingestion transforms raw documents (such as PDFs or webpages) into a queryable format through three steps:

Semantic Chunking: Splitting documents into smaller, meaningful pieces.
Vector Representation: Creating embeddings for each chunk using an embeddings model.
Vector Storage: Loading these chunks and vectors into a specialized vector store.

For this implementation, LangChain handled document loading and splitting. Embeddings were generated using a quantized HuggingFace model via the Transformers.js package, and the vector store was managed by Voy, a Web Assembly-based vector store.

Retrieval and Generation

To answer a user query, the system searches the vector store for document chunks most semantically similar to the input. These chunks, combined with the original question, guide the LLM to produce a final answer.

For follow-up questions containing pronouns or references to previous chat history, a dereferencing step is added. This step rephrases the initial query into a "standalone" question to ensure the vector store retrieval remains accurate.

The Role of Ollama in Local Web Apps

While task-specific models (like embeddings) can run efficiently in the browser, full-scale LLMs are typically too large to be bundled directly into web applications. The author found that browser-based LLM projects often either failed to generate high-quality responses or required multi-gigabyte downloads that caused significant latency.

Ollama solves this by allowing a locally running model to be exposed to a web application via a shell command. In the project described, the Mistral 7B model was used, which ran effectively on a 16GB M2 MacBook Pro. This allows the web app to leverage a powerful, pre-installed LLM on the user's machine without the overhead of shipping the model within the app bundle.

Future Outlook and Browser API Proposals

Local LLM execution is becoming more viable as open-source models become smaller and faster and consumer hardware increasingly includes GPUs. However, because non-technical users are unlikely to run shell commands to configure CORS and start servers, a more seamless integration is required.

The author proposes a new browser API that would allow a web application to request access to a locally running LLM via a popup, enabling the app to use local compute power alongside other in-browser technologies without requiring manual technical configuration.