jasonacox/TinyLLM
Setup and run a local LLM and Chatbot using consumer grade hardware.
What it solves
TinyLLM addresses the challenge of running powerful Large Language Models (LLMs) locally on consumer-grade hardware. It provides a bridge between high-performance AI models and standard home computers, allowing users to maintain privacy and control without needing expensive enterprise infrastructure.
How it works
The project acts as a bridge that connects a local inference engine—such as Ollama, vLLM, or llama-cpp-python—to a user-friendly web interface. These engines serve the model via an OpenAI-compatible API, which the TinyLLM chatbot then consumes. This setup allows you to interact with models through a browser, manage conversation history, and perform tasks like summarizing external websites or fetching real-time data like news and stock prices.
Who it’s for
This project is for developers, researchers, or hobbyists who have access to consumer hardware (such as NVIDIA GPUs or Apple Silicon) and want to host their own private AI chatbot environment without relying on cloud-based services.
Highlights
- Supports multiple popular inference backends including Ollama, vLLM, and llama-cpp-python.
- Provides a ChatGPT-like web interface with support for multi-session chat history.
- Enables Retrieval Augmented Generation (RAG) capabilities, allowing the AI to process PDFs, external URLs, and vector databases.
- Includes built-in tools for real-time data retrieval, such as weather, stock prices, and news headlines.
- Compatible with the
llmCLI tool for command-line interactions with your local model.
Related
- Project
- Project
- Project
- Project