aws/aws-sdk-pandas

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

What it solves

This project simplifies the process of moving data between the popular Python Pandas library and various AWS services. It removes the boilerplate code typically required to interact with AWS data stores, allowing developers to read and write data directly to and from S3, Athena, Redshift, and other services using familiar Pandas syntax.

How it works

It acts as a wrapper around existing AWS SDKs (like Boto3) and Pandas. It provides high-level functions that handle complex operations—such as partitioning data, managing schema evolution, and handling large-scale parallel processing—automatically. It can also integrate with Ray and Modin to distribute data processing tasks across clusters for high-performance workloads.

Who it’s for

Data engineers, data scientists, and developers who work with Python and need to integrate AWS data infrastructure into their data pipelines or analytics workflows.

Highlights

  • Seamless integration with Amazon S3, Athena, Redshift, Glue, DynamoDB, and more.
  • Simplifies complex tasks like writing partitioned Parquet files or querying data lakes.
  • Supports distributed computing at scale via integration with Ray and Modin.
  • Includes specific utilities for modern AI/ML workflows, such as Amazon Bedrock integration and vector data handling.
  • Designed for production environments with support for AWS Lambda, Glue jobs, and SageMaker notebooks.

Related

  • Project
    databricks/databricks-sdk-py

    A Python client library that wraps all Databricks REST APIs, handling authentication (PAT, Azure, GCP, OIDC), retries, pagination, and long‑running operations, enabling programmatic control of clusters, jobs, Unity Catalog, and other Lakehouse resources.

  • Project
    aws/sagemaker-python-sdk

    An open-source library for training and deploying machine learning models on Amazon SageMaker, supporting popular frameworks and foundation model fine-tuning.

  • Project
    cohere-ai/cohere-python

    The Cohere Python SDK is a client library that allows developers to access Cohere's AI models across various platforms including AWS, Azure, GCP, and Oracle OCI.

  • Project
    pandas-dev/pandas

    Pandas is a foundational Python library for data manipulation and analysis, providing high-performance, flexible data structures to handle labeled and relational data.

  • Project
    strands-agents/tools

    A comprehensive library of pre-built tools for AI agents to perform file operations, web automation, system execution, and multi-agent orchestration.