JohnSnowLabs/spark-nlp

Spark NLP (JohnSnowLabs/spark‑nlp)

What it is – An open‑source library that brings state‑of‑the‑art natural‑language‑processing (NLP) and large‑language‑model (LLM) capabilities to Apache Spark. It lets you run modern NLP pipelines (tokenization, NER, sentiment, translation, summarisation, question‑answering, image‑to‑text, speech‑to‑text, etc.) at scale on a Spark cluster, with optional GPU or Apple‑silicon acceleration.

Key points from the README

Scale‑first design – built on Spark, so the same code works on a single laptop or a massive distributed cluster.
Huge model zoo – >100 000 pretrained pipelines and models covering 200+ languages, including BERT, RoBERTa, Llama‑2, Whisper, Vision Transformers, and many more.
Multi‑framework import – can ingest models from TensorFlow, ONNX, OpenVINO, and Llama.cpp (GGUF).
Cross‑language support – usable from Python, R, Java, Scala, Kotlin; packages are published to PyPI, Conda, Maven Central.
Hardware flexibility – separate Maven/pip artifacts for CPU, GPU, AArch64, and Apple‑silicon; sparknlp.start(gpu=True) or apple_silicon=True toggles the runtime.
Broad ecosystem compatibility – works with Databricks, Amazon EMR (including Serverless), Google Dataproc, Zeppelin, Jupyter, Colab, Kaggle, etc.
Offline capability – models and pipelines can be cached locally or stored on S3 for completely offline inference.
Open‑source license – Apache 2.0.

Typical workflow (Python example)

import sparknlp
from sparknlp.base import *
from sparknlp.annotator import *
from sparknlp.pretrained import PretrainedPipeline

spark = sparknlp.start(gpu=True)          # start Spark session with GPU support
pipeline = PretrainedPipeline('explain_document_dl', lang='en')
result = pipeline.annotate("The Mona Lisa is a 16th‑century painting …")
print(result['entities'])   # → ['Mona Lisa', 'Leonardo', 'Louvre', 'Paris']

The example shows how a single line loads a ready‑made pipeline and returns token‑level annotations such as entities, POS tags, embeddings, etc.

Installation

Choose the artifact that matches your Spark version (e.g., pip install spark-nlp==7.0.0 pyspark==3.5.1 for Spark 3.x, or spark-nlp-spark400 for Spark 4.0).
For Scala/Java projects, add the appropriate Maven coordinate (spark-nlp_2.12, spark-nlp-spark400_2.13, etc.).
GPU, AArch64, and Apple‑silicon variants are published under separate package names (spark-nlp-gpu, spark-nlp-aarch64, spark-nlp-silicon).

Who would use it

Data engineers needing NLP at the petabyte scale inside Spark pipelines.
ML practitioners who want to fine‑tune or run pretrained LLMs without leaving the Spark ecosystem.
Enterprises that run on Databricks, EMR, or other managed Spark services and require multilingual, multimodal (text + image + audio) processing.
Researchers who want a unified API for classic NLP tasks (POS, NER) and modern transformer‑based models.

Where to learn more

Official site & docs: https://sparknlp.org/
Model hub with benchmarks: https://sparknlp.org/models
Example notebooks: https://github.com/JohnSnowLabs/spark-nlp/tree/master/examples
Academic citation: Spark NLP: Natural language understanding at scale (Software Impacts, 2021).

All information above is taken directly from the repository’s README; no external assumptions have been added.

JohnSnowLabs/spark-nlp

Spark NLP (JohnSnowLabs/spark‑nlp)

Related