NVIDIA/cudf-spark

NVIDIA cuDF for Apache Spark plugin - accelerate Apache Spark with GPUs

What it solves

It accelerates Apache Spark processing by leveraging GPUs, reducing the time required for data processing tasks within the Spark ecosystem.

How it works

The project provides a plugin library that uses the cuDF (CUDA DataFrame) libraries to offload and accelerate Spark SQL operations on NVIDIA GPUs.

Who it’s for

Data engineers and data scientists who use Apache Spark for large-scale data processing and want to improve performance through GPU acceleration.

Highlights

  • GPU-accelerated processing for Apache Spark via cuDF libraries.
  • Aims for bit-for-bit identical results compared to standard Apache Spark.
  • Provides APIs for zero-copy data transfer into other GPU-enabled applications, such as ML libraries like XGBoost.
  • Includes separate qualification and profiling tools for performance optimization.

Related

  • Project
    NVIDIA/cudf

    GPU‑accelerated DataFrame library that mirrors pandas, Polars, and Dask APIs, enabling fast tabular data cleaning and aggregation for AI/ML workflows.

  • Project
    rapidsai/cugraph

    A collection of GPU-accelerated graph analytics packages that enable the creation and manipulation of graphs and the execution of scalable graph algorithms.

  • Project
    JohnSnowLabs/spark-nlp

    Spark NLP is an Apache Spark‑native library that provides a massive collection of pretrained NLP and LLM models (BERT, Llama‑2, Whisper, Vision Transformers, etc.) for over 200 languages. It supports CPU, GPU, Apple‑silicon, and AArch64 builds, can import models from TensorFlow/ONNX/OpenVINO, and runs on any Spark cluster (Databricks, EMR, GCP Dataproc, etc.). Install via pip or Maven, start a Spark session with `sparknlp.start()`, load a pretrained pipeline, and obtain token‑level annotations such as entities, POS tags, embeddings, translations, or generated text. Licensed under Apache 2.0.

  • Project
    NVIDIA/gds-nvidia-fs

    A kernel driver (`nvidia-fs.ko`) that implements NVIDIA GPUDirect Storage, allowing supported NVMe, computational‑storage, or RDMA filesystems to move data directly into GPU memory, boosting AI/ML data‑pipeline throughput.

  • Project
    NVIDIA/aerial-cuda-accelerated-ran

    NVIDIA Aerial™ CUDA‑Accelerated RAN is an open‑source SDK that implements 5G PHY and MAC processing on NVIDIA GPUs. It includes CUDA kernels for channel coding, modulation, MIMO, and scheduling, plus a Python API for AI/ML research, all packaged in ready‑to‑run Docker containers.