aws-neuron/aws-neuron-sdk

Powering AWS purpose-built machine learning chips. Blazing fast and cost effective, natively integrated into PyTorch and TensorFlow and integrated with your favorite AWS services

What it solves

It provides a high-performance environment for developing, profiling, and deploying deep learning and generative AI workloads specifically for AWS's own AI hardware accelerators (Inferentia and Trainium).

How it works

The SDK includes a graph compiler and runtime that translates machine learning models into a format the hardware can execute. It integrates with popular frameworks like PyTorch and JAX, and supports vLLM Neuron for inference serving. For advanced users, the Neuron Kernel Interface (NKI) allows for the direct programming of NeuronCore for custom kernels.

Who it’s for

Machine learning engineers and developers who are using AWS EC2 instances (such as Inf1, Inf2, Trn1, Trn2, and Trn3) to train or run inference on large-scale AI models.

Highlights

  • High-performance deep learning and generative AI support
  • Integration with PyTorch, JAX, and vLLM
  • Direct hardware programming via the Neuron Kernel Interface (NKI)
  • Dedicated developer tools including Neuron Explorer

Related

  • Project
    huggingface/optimum

    An extension of the Hugging Face ecosystem that provides optimization tools to train and run AI models with maximum efficiency on targeted hardware accelerators.

  • Project
    NVIDIA/TensorRT

    NVIDIA TensorRT is a high-performance deep learning inference library and SDK that accelerates AI model deployment on NVIDIA GPUs.

  • Project
    amd/xdna-driver

    A Linux kernel driver and XRT SHIM library that enables AI inference acceleration on AMD XDNA (NPU) devices for RyzenAI processors.

  • Dispatch
    Accelerate BERT Inference with Hugging Face Transformers and AWS Inferentia

    Hugging Face provides a guide on using AWS Inferentia and the Neuron SDK to accelerate BERT inference, achieving 5-6ms latency on Amazon SageMaker.

  • Dispatch
    Building Blocks for Foundation Model Training and Inference on AWS

    Hugging Face details a four-layer architectural framework on AWS—comprising infrastructure, resource orchestration, ML software stacks, and observability—to support the evolving scaling laws of pre-training, post-training, and test-time compute.