runpod/runpod-python

What it solves

This library provides a unified Python interface to manage RunPod's GPU cloud infrastructure and deploy serverless AI models. It eliminates the need for manual API calls by providing a high-level SDK for both infrastructure management (Pods) and the creation of custom serverless endpoints.

How it works

The project is split into two primary functions:

1. Serverless SDK: Allows developers to wrap a Python function (a handler) and start a worker that can be deployed as a custom API endpoint on RunPod. It includes features like "fitness checks" to validate GPU availability or disk space before processing jobs and a VolumeCache to speed up model loading from network volumes.
2. API Wrapper: A GraphQL wrapper that enables programmatic control over RunPod resources. Users can create, stop, resume, or terminate GPU pods and interact with deployed endpoints using synchronous or asynchronous request methods.

Who it’s for

• AI Engineers: Who need to deploy their models as scalable serverless APIs.
• MLOps Engineers: Who want to automate the provisioning and management of GPU cloud instances (Pods).
• Developers: Who want to programmatically trigger inference requests to RunPod endpoints.

Highlights

• Serverless Worker Framework: Easily turn Python functions into deployable API endpoints.
• Infrastructure Automation: Full programmatic control over GPU and CPU pods.
• Warm Cache: Reduces cold-start times by hydrating local directories from network volumes.
• Fitness Checks: Ensures workers are healthy (e.g., GPU present) before they accept jobs.
• Flexible API Key Management: Supports both global and endpoint-specific keys for thread-safe concurrent operations.