Arm and ExecuTorch 0.7: Expanding Generative AI to Billions of Devices

Arm and the ExecuTorch 0.7 beta are integrating KleidiAI by default to provide automatic AI acceleration for a vast range of Arm CPU architectures. This integration allows Generative AI (GenAI) and Large Language Models (LLMs) to run efficiently on both the latest high-end hardware and devices up to five years old, including the Raspberry Pi 5.

Automatic Acceleration via KleidiAI and ExecuTorch 0.7

ExecuTorch 0.7 beta enables turn-key performance for Android and cross-platform developers by integrating Arm KleidiAI. KleidiAI serves as an AI accelerator layer that embeds into widely-used Edge AI frameworks such as XNNPack, MediaPipe, MNN, ONNX Runtime, and llama.cpp.

By enabling KleidiAI by default in ExecuTorch 0.7, developers gain immediate access to performance optimizations without requiring custom tuning or code changes. This results in:

  • Faster model startups
  • Lower latency
  • Leaner memory footprints

Leveraging SDOT for Broad Hardware Compatibility

To expand GenAI accessibility beyond flagship smartphones, Arm is leveraging the SDOT (Signed Dot Product) instruction. Introduced in the Armv8.2 architecture, SDOT enables efficient dot product operations on vectors of 8-bit signed integers, which is critical for accelerating the matrix multiplication routines that power LLMs using Int8 or Int4 precision formats.

SDOT is widely supported across the Arm ecosystem, appearing in approximately 3 billion devices (roughly 72% of all Arm-based devices). This widespread availability allows models like Llama 3.2 1B to run efficiently on the majority of Android devices and edge hardware like the Raspberry Pi 5.

Performance Benchmarks and Use Cases

High-End Hardware Performance

Previous collaborations between ExecuTorch and KleidiAI focused on Int4 matrix-multiplication using the I8MM feature (Armv8.6 architecture and later). On the Galaxy S24+, this configuration achieved:

  • Prefill performance: Over 20% higher than non-KleidiAI kernels, reaching more than 350 tokens per second.
  • Decode performance: Over 40 tokens per second.

This performance is sufficient for real-time on-device tasks, such as summarizing approximately 600 tokens (roughly 50 unread messages) with a smooth user experience.

Accessibility on Older Hardware

While devices supporting only the SDOT extension may not match flagship speeds, the decode phase often remains faster than average human reading speed. This enables several practical, offline GenAI use cases:

  • Private Smart Assistants: Combining a local LLM with speech-to-text and text-to-speech models for fully offline, voice-based interactions.
  • Context-Aware Text Completion: Real-time, intelligent suggestions in local text editors for writing or coding workflows without an internet connection.

Conclusion

Through the combination of SDOT, KleidiAI, and ExecuTorch 0.7, Arm is extending Generative AI capabilities to billions of existing devices, moving the technology beyond high-end flagships to a broader base of global hardware.

Sources