awslabs/data-on-eks

DoEKS is a tool to build, deploy and scale Data Platforms on Amazon EKS

What it solves

Data on EKS (DoEKS) simplifies the deployment and scaling of data platforms on Amazon EKS. It addresses the complexity of managing the Kubernetes ecosystem and selecting the right configurations for big data workloads, allowing users to quickly build production-ready clusters and conduct proof-of-concepts.

How it works

DoEKS provides a set of opinionated open-source blueprints that integrate various data tools, Kubernetes operators, and AWS Data Analytics managed services. These blueprints automate the deployment of frameworks for distributed data processing, real-time stream processing, and workflow orchestration.

Who it’s for

Data engineers and platform architects who need to run scalable data workloads—such as Spark, Flink, Kafka, and Airflow—on Kubernetes using Amazon EKS.

Highlights

  • Data Processing: Blueprints for Apache Spark, Amazon EMR on EKS, and Ray.
  • Streaming Platforms: Support for Apache Flink and Apache Kafka (via Strimzi).
  • Orchestration: Deployment patterns for Apache Airflow and Argo Workflows.
  • Databases & Query Engines: Integration with Trino, Pinot, and ClickHouse.

Related

  • Project
    aws/sagemaker-spark

    An open-source Spark library that integrates Amazon SageMaker into Spark ML pipelines for scalable model training and inference on DataFrames.

  • Project
    fluid-cloudnative/fluid

    A Kubernetes-native distributed dataset orchestrator and accelerator that optimizes data access for AI and big data applications by providing dataset abstraction and elastic caching.

  • Project
    qiantongtech/qData

    An open-source data middle platform for enterprise data governance and development, featuring ETL, metadata management, and AI-powered Text2SQL data Q&A.

  • Project
    kubeflow/community-distribution

    A distribution of manifests for installing the Kubeflow Platform on various Kubernetes clusters, providing a complete end-to-end machine learning environment.

  • Project
    platformplatform/PlatformPlatform

    A comprehensive SaaS blueprint for B2B/B2C products featuring a .NET 10 and React 19 stack, integrated Stripe billing, and an AI-native multi-agent development workflow.