databricks-industry-solutions/pixels

Facilitates simple large scale processing of HLS Medical images, documents, zip files. OHIF Viewer, 2 segmentation models and interactive learning.

What it solves

Medical imaging data, specifically DICOM files, are often difficult to analyze at scale because they are stored as unstructured files rather than in databases. This project bridges that gap by turning millions of medical images into structured, SQL-accessible metadata, enabling hospitals and researchers to perform large-scale analytics, AI model training, and secure visualization.

How it works

The solution uses a specialized library to ingest DICOM files from cloud storage, extract their technical metadata, and store it in Databricks tables. It integrates with the OHIF medical viewer to allow users to label images, trigger AI-assisted segmentation using NVIDIA’s MONAI framework, and run real-time inference on GPU-enabled clusters. All data access is governed by Unity Catalog, ensuring that sensitive patient information is protected through built-in anonymization features.

Who it’s for

This tool is designed for healthcare data engineers, radiologists, and medical AI researchers who need to build scalable pipelines for medical imaging, automate image segmentation, or create dashboards for clinical research using the Databricks Data Intelligence Platform.

Highlights

  • Scalable Ingestion: Processes millions of DICOM files using Spark, with support for streaming and automatic unzipping.
  • AI-Ready: Built-in integration with NVIDIA MONAI for automated image segmentation and custom model training.
  • Secure Visualization: Includes a web-based DICOM viewer (OHIF) that respects data access controls and supports NIfTI segmentation overlays.
  • Privacy-First: Provides automated metadata anonymization using format-preserving encryption to ensure compliance.
  • SQL-Accessible: Converts complex image headers into standard tables, allowing for easy cohort building and dashboarding via SQL.

Related

  • Project
    databricks/databricks-sdk-py

    A Python client library that wraps all Databricks REST APIs, handling authentication (PAT, Azure, GCP, OIDC), retries, pagination, and long‑running operations, enabling programmatic control of clusters, jobs, Unity Catalog, and other Lakehouse resources.

  • Project
    databricks-demos/dbdemos

    dbdemos is a pip‑installable Python toolkit that automates the setup of Databricks Lakehouse demo bundles—installing notebooks, clusters, pipelines, dashboards and optional ML models with a single command.

  • Project
    pipeshub-ai/pipeshub-ai

    PipesHub is an open‑source, self‑hostable platform that connects enterprise data sources (Slack, Google Drive, GitHub, etc.) to AI applications. It builds a permission‑aware knowledge graph, stores vector embeddings, and lets any LLM generate explainable answers with citations. The project includes Docker‑Compose deployment, a no‑code agent builder, MCP support, and SDKs for Python, TypeScript, and Go.

  • Project
    abstractqqq/polars_ds_extension

    PDS is a Python package that extends the Polars dataframe library with data-science features: parallel ML metrics, quick on-the-fly regression models, string/array distance functions, statistical tests, convenience aggregations, a reusable data-transformation pipeline, nearest-neighbor queries, and compatibility with Pandas dataframes. It's installed via `pip install polars-ds` and requires Polars ≥ 1.4.0.

  • Project
    Trampoline-AI/predict-rlm

    predict-rlm is a Python library that gives large language models a sandboxed, stateful runtime. Models write and execute Python code, keep mutable state, call focused sub‑models, and use tools/skills (e.g., PDF handling, spreadsheets). The framework records a full trace for observability and can feed those traces to optimisation tools like GEPA. It’s aimed at complex, multi‑step workflows such as document analysis, audit, or spreadsheet automation, and supports optional Docker sandboxing and rich callbacks.