aws/aws-sdk-pandas

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

What it solves

AWS SDK for pandas (formerly awswrangler) simplifies the process of moving and managing data between pandas DataFrames and various AWS services. It removes the need to write complex boilerplate code when integrating pandas with AWS data lakes, databases, and analytics services.

How it works

The library provides a set of high-level Python functions that wrap AWS SDKs to allow direct reading and writing of data. It supports a wide range of AWS services including S3 (Parquet, CSV, JSON, Excel), Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, and DynamoDB, as well as traditional databases like PostgreSQL, MySQL, and SQL Server.

Who it’s for

Data engineers, data scientists, and analysts who use pandas for data manipulation and need to efficiently ingest, store, and query data across the AWS ecosystem.

Highlights

  • Broad AWS Integration: Direct support for a vast array of AWS services including S3, Athena, Glue, and Redshift.
  • Scalability: Ability to run workflows at scale by integrating with Modin and Ray to distribute processing over worker clusters.
  • Flexible Data Formats: Seamlessly handles Parquet, CSV, JSON, and Excel files on S3.
  • Comprehensive API: Includes specialized tools for schema evolution, partition projection, and data quality checks via Glue Data Quality.

Related

  • Project
    runpod/runpod-python

    The official Python library for RunPod, providing an SDK to build serverless AI workers and a GraphQL wrapper to manage GPU cloud pods and endpoints.

  • Project
    databricks/databricks-sdk-py

    A Python SDK that provides a robust, typed interface for all public Databricks REST API operations to accelerate development for the Databricks Lakehouse.

  • Project
    pandas-dev/pandas

    A powerful Python data analysis toolkit that provides fast, flexible, and expressive data structures for working with relational or labeled data.

  • Project
    Kaggle/kagglehub

    A Python library for programmatically downloading, uploading, and loading Kaggle datasets, models, and competition resources into data science frameworks.

  • Project
    aws/sagemaker-python-sdk

    An open-source library for training and deploying machine learning models on Amazon SageMaker, supporting popular frameworks and foundation model fine-tuning.