Ascend/pytorch

What it solves

TorchNPU allows the PyTorch deep learning framework to run directly on Huawei Ascend AI processors (NPUs). It bridges the gap between PyTorch's high-level APIs and the specific hardware architecture of Ascend NPUs, enabling developers to leverage Ascend's compute power without abandoning the PyTorch ecosystem.

How it works

It acts as an adapter plugin that integrates with PyTorch via a mechanism called PrivateUse1. It provides several core functional modules:

• Basic Compute: Implements PyTorch native and custom APIs to support mainstream AI workloads.
• Distributed Training: Accelerates large-scale training using FSDP2 and supports collective communication primitives (like AllReduce and AllGather) and DTensor.
• Graph Mode: Uses a combination of dynamic graph capture, static graph optimization, and efficient code generation to speed up training and inference.
• Memory Management: Employs virtual memory management to reduce fragmentation and optimizes memory reuse across streams in distributed settings.
• Debug & Tuning: Includes tools for profiling compute, memory, and communication, along with a "WatchDog" for monitoring communication anomalies.

Who it’s for

AI developers and researchers who use PyTorch and want to migrate their models to or train them on Huawei Ascend NPU hardware for improved performance.

Highlights

• Seamless Integration: Inherits mature PyTorch capabilities while adding NPU-specific optimizations.
• Distributed Scaling: Dedicated support for FSDP2 and DTensor for high-performance parallel training.
• Performance Optimization: Offers a Graph Mode for accelerated execution via NPUGraph.
• Hardware-Specific Tuning: Provides deep profiling and real-time communication monitoring.