databricks/databricks-sdk-py

Databricks SDK for Python (Beta)

What it is – A Python client library that wraps the entire public Databricks REST API. It lets you programmatically manage Databricks workspaces, clusters, jobs, Unity Catalog, and other Lakehouse resources from Python code.

Why it matters – Databricks is a widely‑used cloud data‑lake/warehouse platform that also hosts Spark‑based machine‑learning workloads. The SDK abstracts the raw HTTP calls, adds intelligent retries, pagination handling, and unified authentication, making it much easier to build automation, CI/CD pipelines, or custom tools that interact with Databricks.

Getting started

pip install databricks-sdk

from databricks.sdk import WorkspaceClient
w = WorkspaceClient()          # picks up host/token from .databrickscfg, env vars, or defaults
for c in w.clusters.list():
    print(c.cluster_name)

The SDK works with Python 3.7‑3.11. In Databricks Runtime 13.1+ the library is pre‑installed; otherwise upgrade with %pip install --upgrade databricks-sdk inside a notebook and restart the Python process.

Key capabilities (as described in the README)

• Full API coverage – every public Databricks REST endpoint is exposed as a Python method.
• Robust HTTP client – automatic retries and back‑off for transient failures.
• Paginated responses – iterator helpers hide pagination details.
• Long‑running operation handling – helpers to wait for jobs, clusters, etc., to reach a desired state.
• Unified authentication – supports:
  • Databricks personal‑access‑token (PAT) authentication
  • Workload Identity Federation (OIDC) tokens
  • Azure native flows (client‑secret, Azure CLI, Managed Service Identity)
  • GCP native flows (service‑account JSON, default application credentials)
  • Automatic detection of the cloud provider (AWS, Azure, GCP) and selection of the appropriate method.
• Configuration sources – credentials can be supplied via function arguments, environment variables, or a ~/.databrickscfg profile file. The SDK searches these locations in a defined order and stops at the first valid set.
• Unified host support – a single host can serve both account‑level and workspace‑level APIs when account_id and workspace_id are present, allowing the same profile to create both WorkspaceClient and AccountClient objects.
• User‑agent attribution – the client adds identifiable headers for Databricks to track SDK usage.
• Error handling & logging – consistent exception types and optional logging hooks.

Example use‑cases shown in the repo

• OAuth flow from a Flask web server (examples/flask_app_with_oauth.py).
• Starting a job and waiting for it to finish (examples/starting_job_and_waiting.py).
• Local‑browser OAuth login (examples/local_browser_oauth.py).
• Real‑world projects such as Unity Catalog migration and IP‑access‑list analysis rely on this SDK.

Where to learn more

• Full API reference: https://databricks-sdk-py.readthedocs.io
• Cloud‑specific docs: AWS, Azure, GCP links in the README.
• Related SDKs: Java, Go, and the Terraform provider.

License – Open‑source under the same license as the upstream Databricks repository (see LICENSE).