amaiya/onprem

What it solves

OnPrem.LLM is a privacy-conscious toolkit designed for applying large language models to sensitive, non-public data. It allows users to run LLMs in offline or restricted environments, ensuring that data remains local by default while still providing the flexibility to integrate with cloud-based providers when needed.

How it works

The toolkit provides a unified Python interface to interact with various LLM backends. It supports local execution via llama-cpp-python, Hugging Face transformers, Ollama, and vLLM, as well as cloud providers like OpenAI and Anthropic. It includes built-in support for Retrieval-Augmented Generation (RAG) for document ingestion and questioning, structured outputs using Pydantic, and a sandboxed environment for executing AI agents.

Who it’s for

It is intended for developers and organizations that need to process sensitive documents using LLMs without sacrificing privacy or security, particularly those operating in restricted or air-gapped environments with modest computational resources.

Highlights

• Local-first approach: Fully local execution with optional cloud integration.
• Versatile backend support: Compatible with llama-cpp-python, transformers, transformers, vLLM, and LiteLLM-supported providers.
• Document Intelligence: Built-in pipelines for information extraction, summarization, classification, and question-answering.
• Sandboxed Agents: An AgentExecutor that allows AI agents to be launched safely in a restricted environment.
• Visual Workflow Builder: A point-and-click interface for assembling complex document analysis pipelines.
• Resource Efficient: Includes modules like SparseStore for RAG without requiring pre-stored embeddings.