graphframes/graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

What it solves

GraphFrames addresses the need for scalable graph processing and analytics within the Apache Spark ecosystem. It allows users to perform complex graph computations on massive datasets by integrating graph-specific algorithms with the familiar, high-performance DataFrame API.

How it works

It is built on top of Apache Spark and uses the DataFrame abstraction to represent graphs. Users define a graph by providing two DataFrames: one for vertices (nodes) and one for edges (relationships). This structure allows you to run built-in distributed algorithms (like PageRank or connected components) or create custom processing logic using APIs like Pregel and AggregateMessages. Because it leverages the Spark SQL engine, it benefits from built-in performance optimizations while enabling users to mix graph-based analysis with standard relational queries.

Who it’s for

It is designed for data engineers, data scientists, and developers working with large-scale network data who need to perform tasks like fraud detection, identity resolution, social network analysis, or knowledge graph construction using Java, Scala, or Python.

Highlights

  • Scalable graph algorithms including PageRank, connected components, and label propagation.
  • Powerful motif finding API for querying complex patterns within network data.
  • Flexible custom processing via Pregel and AggregateMessages interfaces.
  • Seamless integration with Apache Spark DataFrames for high-performance data manipulation.
  • Support for Property Graph models to store and analyze rich metadata on nodes and edges.

Related

  • Project
    scverse/gget

    A command-line tool and Python package for efficient, single-line querying of genomic reference databases and sequence analysis.

  • Project
    tmwgsicp/wechat-download-api

    An open-source API and RSS generator for WeChat Official Accounts that enables full-content scraping, multi-format archiving, and AI client integration via MCP.

  • Project
    mlmed/torchxrayvision

    TorchXRayVision is an open‑source Python library that standardizes access to many public chest‑X‑ray datasets and supplies pretrained deep‑learning models (DenseNet‑121, ResNet‑50, auto‑encoders, segmentation nets). It lets researchers load images, metadata, and pathology labels with a uniform API, apply built‑in preprocessing, and obtain pathology probabilities instantly, facilitating rapid prototyping, benchmarking, and few‑shot learning in medical imaging.

  • Project
    Azure/AzurePublicDataset

    A collection of public, production-scale traces from Microsoft Azure that provides researchers with real-world data on cloud workloads, serverless computing, AI inference, and agentic coding sessions.