dotnet/spark

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

What it solves

.NET for Apache Spark allows developers to use the power of Apache Spark—a popular engine for big data processing—directly within the .NET ecosystem. It bridges the gap between .NET languages and the Spark runtime, enabling C# and F# developers to perform large-scale data analysis without needing to switch to Java, Scala, or Python.

How it works

It provides high-performance APIs that act as a wrapper around the Apache Spark engine. When you run an application, it uses a .NET Worker and a Scala bridge to communicate with the Spark runtime. It leverages tools like Apache Arrow for efficient data transfer between the Java Virtual Machine (JVM) and the .NET environment, allowing you to use DataFrames, Spark SQL, and Structured Streaming in your .NET code.

Who it’s for

This project is for data engineers and software developers who are already proficient in C# or F# and need to process large datasets or streaming data using Apache Spark on platforms like Windows, Linux, or macOS.

Highlights

  • Supports modern .NET versions (including .NET 8) and Apache Spark 3.x through 4.x.
  • Enables use of DataFrames and Spark SQL for structured data processing.
  • Supports Structured Streaming for real-time data analysis.
  • Provides deployment guides for major cloud platforms like Azure HDInsight, Amazon EMR, and Databricks.
  • Offers native C# and F# examples for common tasks like batch processing and Kafka streaming.

Related

  • Project
    TileDB-Inc/TileDB

    TileDB is a universal storage engine for dense and sparse multi-dimensional arrays, enabling efficient storage and access for complex data science workloads.

  • Project
    Syngnat/GoNavi

    GoNavi is a lightweight, native‑binary database GUI built with Go + Wails and React. It supports dozens of data sources, offers a Monaco‑based SQL editor, virtualised result grids, and schema‑aware AI assistants (OpenAI, Gemini, Claude, etc.) that can generate or optimise queries. The app exposes an MCP HTTP interface for external agents, runs in ~20‑26 MB installers, and is Apache‑2.0 licensed.

  • Project
    peteromallet/dataclaw

    DataClaw is a macOS app and Python CLI that extracts, redacts, and publishes coding‑agent conversation logs (Claude Code, Codex, etc.) as Hugging Face datasets, creating a community‑sourced corpus of human‑AI coding interactions.

  • Project
    apache/tinkerpop

    Apache TinkerPop is a graph computing framework that provides a universal language and interface for interacting with various graph databases, preventing vendor lock-in.