ggml 1.0.0 release notes
Overview
ggml 1.0.0 marks a significant milestone for the tensor library, focusing on API stabilization, performance tuning, and expanded hardware support. This release prunes deprecated public APIs and legacy operations, introduces configurable YaRN truncation, and adds FP8 support. Performance improvements are widespread, spanning backends from CUDA and Metal to Vulkan, SYCL, and OpenCL, ensuring that local inference remains efficient across diverse hardware configurations.
Core and API Enhancements
The 1.0.0 release prioritizes modernization and efficiency in the core library:
- API Cleanup: Deprecated public APIs and legacy
DIAG_MASKoperations have been removed. The RPC protocol has been bumped to v9, improving loading times and implementing more graceful failure handling for unreachable endpoints. - YaRN Truncation: New APIs (
ggml_rope_set_truncateandggml_rope_get_truncate) allow for precise control over YaRN correction-bound rounding. - Scheduler Improvements: The compute scheduler now skips uploading unused inputs and prefetches only those produced by earlier splits, reducing overhead. Additionally, the Mixture-of-Experts (MoE) cache can now compute cache misses asynchronously on the CPU, preventing total failure when models exceed GPU memory budgets.
Backend Performance Updates
Performance gains are a hallmark of this release, with specific optimizations for major compute platforms:
- CUDA: Added few-row Matrix Multiplication Quantization (MMQ) for batch sizes 2-24 and Matrix-Vector-Fused (MMVF) for BF16 matmuls. Support for OCP FP8 and fused NVFP4 gate/up matmuls has been implemented, alongside improved graph reuse logic.
- Metal: Fused MoE routing and optimized command buffer management have been introduced to improve throughput on Apple Silicon.
- Vulkan: Added an Intel FlashAttention prefill kernel and enabled dequantized KV with CM2. Support for large-tile split-K on RDNA3/4 architectures improves performance on modern AMD hardware.
- SYCL & OpenCL: SYCL now features a device-side MoE scheduler, while OpenCL includes decomposed Adreno FlashAttention and improved multi-column GEMV kernels for mobile chipsets.
- Hexagon & CPU: Expanded Hexagon op coverage and added optimizations for SpacemiT RVV and ARM NEON architectures, ensuring broad support for mobile and embedded processors.
Community Insights
Users of the library highlight the consistent performance gains delivered by the project. Long-time contributors and users note that the library frequently achieves 5-10% speed improvements on existing hardware through iterative optimization.
"What is really impressive about this project and llama.cpp as well is that they keep finding new ways to speed up the code on the same hardware. Every time I upgrade (once a month or so) there is usually a 5 to 10% speed boost." — @jacquesm
While the project is widely praised for its engineering, some community discussion has centered on the evolution of the project's architecture. Users have noted that the library's ability to handle MoE models that exceed VRAM by offloading to host memory is a critical feature for consumer-grade hardware.
Sources
Related
- Dispatch
- Dispatch
- Dispatch
- Dispatch
- Dispatch