projectglow/glow

What it solves

It enables large-scale genomic analysis at biobank-scale and beyond by bridging the gap between bioinformatics and the Spark ecosystem. It allows researchers to handle massive genomic datasets that would otherwise be too large for traditional tools.

How it works

Glow integrates genomic data processing into Apache Spark, allowing genomic files (such as VCF, BGEN, and Plink) to be loaded into distributed DataFrames. Users can then perform queries and analyses using Spark SQL APIs in Python, SQL, R, Java, and Scala. It also provides tools to parallelize existing command-line tools and Pandas functions to scale workflows.

Who it’s for

Bioinformaticians and data scientists working with massive genomic datasets, electronic health records, and medical images who need the scalability of a distributed computing engine.

Highlights

• Supports loading VCF, BGEN, and Plink files into distributed DataFrames.
• Built-in functions for quality control, data manipulation, variant normalization, and liftOver.
• Ability to perform genome-wide association studies (GWAS).
• Integration with Spark ML libraries for population stratification.
• Support for multiple languages including Python, SQL, R, Java, and Scala.