amaiya/onprem

A toolkit for applying LLMs to sensitive, non-public data in offline or restricted environments

OnPrem.LLM is a Python toolkit for applying large language models (LLMs) to sensitive, non-public data in offline or restricted environments. It is designed to run fully locally by default, but also supports cloud LLM providers (OpenAI, Anthropic, etc.) when needed. The project is inspired by privateGPT and focuses on document intelligence tasks like information extraction, summarization, classification, question-answering, and AI agents.

Key features include:

  • Local-first execution with optional cloud integration.
  • RAG (Retrieval-Augmented Generation) for question-answering over your own documents, including a lightweight "SparseStore" that avoids pre-storing embeddings, making it usable on modest hardware.
  • Support for many LLM backends: llama.cpp, Hugging Face Transformers, Ollama, vLLM, OpenAI, Anthropic, and AWS GovCloud Bedrock.
  • Structured outputs via Pydantic models.
  • Sandboxed AI agents (AgentExecutor) that can safely perform tasks like file analysis.
  • A visual workflow builder for assembling complex document analysis pipelines.
  • Lightweight installs: the base package no longer pulls in PyTorch; optional extras add capabilities like local embeddings, document parsing, or the web UI.

In practice, you can start with a simple prompt, ingest documents for RAG, switch between local and cloud models, and even run agents—all through a consistent Python API. The project emphasizes privacy and flexibility, making it suitable for enterprises or individuals who need to process confidential data without sending it to external services.

Related

  • Project
    PromtEngineer/localGPT

    A private, on-premise document intelligence platform that uses hybrid search and local LLMs to let users query and summarize their files without data leaving their machine.

  • Project
    Trampoline-AI/predict-rlm

    predict-rlm is a Python library that gives large language models a sandboxed, stateful runtime. Models write and execute Python code, keep mutable state, call focused sub‑models, and use tools/skills (e.g., PDF handling, spreadsheets). The framework records a full trace for observability and can feed those traces to optimisation tools like GEPA. It’s aimed at complex, multi‑step workflows such as document analysis, audit, or spreadsheet automation, and supports optional Docker sandboxing and rich callbacks.

  • Project
    janhq/jan

    An open-source ChatGPT alternative that lets users run LLMs locally for privacy and control, or connect to cloud-based AI providers.

  • Project
    Mintplex-Labs/anything-llm

    An all-in-one AI application that enables users to build a private, fully-featured ChatGPT experience with built-in RAG, AI agents, and multi-user support.

  • Project
    simonw/llm

    A CLI tool and Python library for interacting with various remote and local Large Language Models, featuring SQLite logging and plugin-based extensibility.