JohnSnowLabs/spark-nlp

What it solves

Spark NLP is designed to provide high-performance, accurate Natural Language Processing (NLP) annotations for machine learning pipelines that can scale across distributed environments. It addresses the challenge of running state-of-the-art NLP models at scale in production, particularly for those using the Apache Spark ecosystem.

How it works

Built natively on top of Apache Spark, the library extends Spark's distributed processing capabilities to NLP tasks. It allows users to create pipelines of annotators to process text, images, and speech. It supports a vast array of pre-trained models (over 100,000) and provides native support for importing models from TensorFlow, ONNX, OpenVINO, and Llama.cpp (GGUF).

Who it’s for

It is intended for data engineers and machine learning practitioners who need to perform NLP at scale, supporting multiple languages (200+) and multiple programming languages including Python, Scala, Java, and Kotlin.

Highlights

• Massive Model Hub: Access to over 100,000 pre-trained pipelines and models in 200+ languages.
• Distributed Scaling: Native integration with Apache Spark for seamless scaling in clusters.
• Broad Task Support: Covers everything from tokenization and NER to machine translation, summarization, and LLMs like Llama-2, Mistral, and Phi.
• Cross-Platform Compatibility: Works across Python, JVM (Java, Scala, Kotlin), and platforms like Databricks, EMR, and GCP Dataproc.
• Multimodal Capabilities: Includes support for image classification, image-to-text (captioning), and automatic speech recognition.