Real-Time AI Sound Generation on Arm CPUs

On-Device AI Audio Generation for Creative Workflows

Arm has demonstrated a personal sound generation application that enables the creation of studio-ready sounds from text prompts directly on Arm-based CPUs. By leveraging open-source generative AI models and efficient on-device inference, the tool allows music producers to generate unique .wav files in seconds without relying on cloud resources or dedicated GPUs.

Technical Architecture and Model Integration

The sound generation engine is built on a combination of open-source frameworks and optimized hardware execution. The system utilizes the following core components:

  • Model: The Stable Audio Open model by Stability AI, sourced via Hugging Face.
  • Inference Frameworks: PyTorch and TorchAudio are used for model inference and audio signal handling.
  • Hardware Execution: The pipeline runs natively on Arm-based CPUs, utilizing optimized multithreaded execution to maintain responsiveness during diffusion runs.

While the system is optimized for Arm CPUs, it maintains device flexibility and can be configured to run on Metal (Apple Silicon) or CUDA (NVIDIA) by adjusting the device target in the code.

Performance Optimization for Arm CPUs

To achieve real-time performance on CPU-based hardware, the implementation employs specific optimization strategies to reduce latency and memory overhead:

Thread Utilization

To maximize the compute power of the Arm CPU, the application enables full thread utilization using the following PyTorch command:

torch.set_num_threads(os.cpu_count())

Memory Management

To prevent memory leaks and maintain stability across multiple generations, the system periodically triggers garbage collection:

if gen_count % 3 == 0:
    gc.collect()

Inference Tuning

The generation loop is tuned for speed by reducing the step count for the diffusion process, which accelerates inference without sacrificing necessary quality for sound effects:

output = generate_diffusion_cond(
    model,
    steps=7, # Reduced step count for faster inference
    cfg_scale=1,
    # ... other parameters
    sampler_type="dpmpp-3m-sde",
    device=device
)

Integration with Digital Audio Workstations (DAW)

The tool is designed to integrate directly into the music production workflow by outputting generated .wav files to a project folder monitored by Ableton Live. This allows a creator to enter a prompt (e.g., "analog bassline" or "cinematic riser") and a tempo, and immediately see the resulting audio file appear in the Ableton browser for further arrangement and modulation.

Implications for On-Device AI

This prototype demonstrates the viability of efficient, on-device AI inference on Arm CPUs for high-compute creative tasks. The primary benefits of this approach include:

  • Reduced Latency: Eliminating cloud inference removes the wait times associated with network requests.
  • Privacy and Ownership: On-device generation ensures that data remains local and the creator retains full ownership of the outputs.
  • Edge Deployment: The ability to run these models on Arm CPUs extends generative AI capabilities into edge devices and directly into professional creative software interfaces.

Sources