RunanywhereAI/wally

What it solves

Wally is a terminal-based tool that allows users to run open-source AI models locally on their own hardware or use hosted versions when their local machine lacks the resources. It simplifies the process of downloading, managing, and interacting with a wide variety of AI modalities, including text, vision, speech, and embeddings, without requiring complex configuration or API keys for local use.

How it works

Wally acts as a unified interface for multiple AI backends (such as llama.cpp, MLX, Sherpa-ONNX, and ONNX Runtime). It automatically selects the best available engine based on the hardware (e.g., Apple Silicon GPU, Windows ARM64 NPU) and the model format. Users can pull models from a catalog or Hugging Face GGUF files and run them via simple CLI commands. For hosted models, it integrates with a RunAnywhere console for metered access.

Who it’s for

It is designed for developers and power users who want a fast, privacy-focused way to run LLMs and other AI models from the terminal, as well as those who want to integrate AI models into coding tools like Claude Code, Claude Desktop, and JetBrains IDEs (CLion, RustRover).

Highlights

• Multi-modal support: Handles LLMs, Vision-Language Models (VLM), Text-to-Speech (TTS), Speech-to-Text (STT), embeddings, and image generation.
• Hybrid Local/Cloud: Seamlessly switches between local execution and hosted models via a single CLI.
• Hardware-specific optimization: Leverages specialized hardware like Apple's Metal/MLX and Qualcomm's Hexagon NPU.
• Tool Integration: One-command setup to wire models into popular coding agents and IDEs.
• OpenAI-compatible API: Can serve models locally as an HTTP endpoint for other applications.