Shoehorn: Precision-Tuned Model Quantization for Local Hardware

Shoehorn is a tool designed to maximize the utility of local hardware by quantizing language models to fit exactly within a user's available memory budget. Unlike standard preset quantizations, which often leave unused memory or fail to load due to insufficient space, Shoehorn calculates a per-tensor mixed-precision assignment to utilize up to 99.99% of the available memory budget.

Memory-Centric Quantization Strategy

Shoehorn shifts the quantization process from picking a preset to starting with the hardware's actual memory capacity. The tool subtracts the memory required for inference and then solves a mixed-precision assignment for each tensor to ensure the model fits within the the remaining budget. This approach allows for the highest possible model quality given the specific hardware constraints of the machine.

Installation and Backend Requirements

Shoehorn requires llama.cpp to be installed and available on the system PATH to serve as the inference backend. Users can install the tool via Homebrew or from source using Cargo:

  • Homebrew: brew install notactuallytreyanastasio/shoehorn/shoehorn
  • Source: cargo install --path . after cloning the repository.

User Interface and Discovery

Shoehorn includes a local web application that automates the memory measurement and quantization process. The UI provides a "tape measure" budget gauge to visualize memory usage, displays the perplexity cost of the specific fit, and offers a discovery feature that scans Hugging Face's most-downloaded models to rank them based on the quality they can achieve within the user's specific memory budget.

Community Feedback and Technical Observations

While Shoehorn aims for high precision in memory allocation, early user reports indicate potential discrepancies between calculated fits and actual runtime requirements. One user reported receiving an "insufficient memory error" when starting the server despite the tool's sizing results suggesting the model would fit.

Other community discussions have compared the tool to existing projects such as llmfit and questioned its compatibility with other memory-optimization techniques like AirLLM or specific integration possibilities with projects like Colibri.

Sources

Related

  • Dispatch
    Optimizing Local LLM Selection: An In-Depth Look at whichllm

    Explore how whichllm solves the challenge of finding the best-performing local LLM for specific hardware using evidence-based ranking and VRAM estimation.

  • Project
    DaoyuanLi2816/can-i-finetune-this

    A CLI tool to estimate VRAM usage, benchmark performance, and generate training recipes for fine-tuning LLMs on consumer GPUs using LoRA and QLoRA.

  • Project
    AlexsJones/llmfit

    A terminal tool that detects your hardware and recommends the best-fitting LLM models based on memory, speed, quality, and context.

  • Project
    signerless/llm-checker

    An AI-powered CLI tool that analyzes system hardware to recommend and verify the optimal local LLM models for performance and compatibility.

  • Project
    Andyyyy64/whichllm

    A hardware-aware LLM recommendation tool that ranks the best HuggingFace models based on your specific GPU/CPU/RAM and real-world benchmark performance.