sambanova/bloomchat

This repo contains the data preparation, tokenization, training and inference code for BLOOMChat. BLOOMChat is a 176 billion parameter multilingual chat model based on BLOOM.

What is BLOOMChat?

BLOOMChat is a 176‑billion‑parameter, multilingual chat model that builds on the open‑source BLOOM‑176B language model. SambaNova fine‑tuned BLOOM with instruction‑following data (OpenChatKit OIG, Dolly 2.0, and OpenAssistant OASST1) to make it behave like a conversational assistant that can answer questions, generate text, and handle many languages.

What does the repository contain?

Area What you’ll find in the repo
Data preparation Scripts (data_prep/) that further clean and combine the three source datasets into two JSONL files ready for tokenisation.
Tokenisation Bash wrappers (tokenization_prep/) that call SambaNova’s generative_data_prep utilities to turn the JSONL files into HDF5 token streams and split files.
Training Reference scripts (training/) that were used on SambaNova’s in‑house Reconfigurable Dataflow Unit (RDU). They show the hyper‑parameters and the two‑stage fine‑tuning process (first on OIG, then on Dolly 2.0 + OASST1).
Inference (GPU) Step‑by‑step guide to run the released model on standard GPUs using the Hugging‑Face transformers-bloom-inference repo, with small patches to the code and example command lines for bf16 and int8 precision.
Inference (RDU) A pointer to the rdu_quick_start/ folder that contains code for running the model on SambaNova’s RDU via the SambaFlow SDK.
Metadata Links to the model on Hugging Face, a blog post, Discord community, and a citation block.

Who might use this?

Audience Why it matters
Researchers Provides a reproducible pipeline (data prep → tokenisation → training) for a very large multilingual LLM, plus the exact hyper‑parameters used on an RDU.
Developers Gives ready‑to‑run inference scripts for both GPU (with transformers) and SambaNova hardware, plus suggested generation settings (temperature 0.8, top‑p 0.9, etc.).
ML engineers Shows how to integrate SambaNova’s generative_data_prep and SambaFlow SDKs into a training‑inference workflow.
Community members Access to a Discord channel and a Hugging Face Space where you can chat with the model directly.

How to get started (quick‑start summary)

  1. Clone the repo and install the required Python packages (pip install datasets plus the dependencies listed in the Pipfile).
  2. Prepare data – run the two Python scripts in data_prep/ to produce oasst1_dolly.jsonl and bloom_ock_100K_each.jsonl.
  3. Tokenise – point the two bash scripts in tokenization_prep/ at a local checkout of SambaNova’s generative_data_prep repo; they output HDF5 token files under *_out/.
  4. Training – the repo only contains the RDU training scripts (they won’t run on a normal GPU). They are useful for understanding the two‑stage fine‑tuning schedule.
  5. GPU inference – clone huggingface/transformers-bloom-inference, apply the two small diffs shown in the README, then run one of the example commands, e.g.:
    python -m inference_server.cli \
  --model_name sambanovasystems/BLOOMChat-176B-v1 \
  --model_class AutoModelForCausalLM \
  --dtype bf16 \
  --deployment_framework hf_accelerate \
  --generate_kwargs '{"do_sample": true, "temperature": 0.8, "repetition_penalty": 1.2, "top_p": 0.9, "max_new_tokens": 512}'
  6. RDU inference – follow the rdu_quick_start/ instructions and use the SambaFlow SDK if you have access to a SambaNova RDU.

What makes BLOOMChat special?

  • Size & multilinguality – 176 B parameters, able to generate in many languages.
  • Instruction‑tuned – The model has been adapted for chat‑style interactions rather than raw text continuation.
  • Hardware‑aware – Training leveraged SambaNova’s proprietary RDU architecture; the repo documents the exact workflow for that environment.
  • Open‑source transparency – All data‑pre‑processing, tokenisation, and (partial) training scripts are publicly available, and the final model weights are hosted on Hugging Face.

Where to go next?

  • Play with the model on the Hugging Face Space linked in the README.
  • Join the community on Discord for support and updates.
  • Explore the data – the prepared JSONL files and tokenised HDF5 directories can be inspected to understand the instruction data.
  • Adapt the pipeline – replace the source datasets with your own domain data and follow the same steps to fine‑tune a large LLM on an RDU or, with modifications, on GPU clusters.

All statements above are taken directly from the repository’s README; no additional features have been inferred.

Related

  • Dispatch
    BLOOM 176B Training Technology Overview

    Hugging Face released a detailed breakdown of how the 176‑billion‑parameter BLOOM model was trained using 384 A100 GPUs, Megatron‑DeepSpeed 3D parallelism, and BF16 mixed‑precision, highlighting the hardware, software stack, and engineering challenges.

  • Project
    OptimalScale/LMFlow

    LMFlow is an extensible Python toolbox for fine‑tuning large language models. It supports full‑parameter training, LoRA, the memory‑efficient LISA algorithm, QLoRA quantization, and a suite of speed‑up tricks (FlashAttention‑2, DeepSpeed, vLLM, SGLang). After training, it provides simple scripts for chat‑style inference and optional Gradio/Flask deployment. The library is designed for Linux/Colab environments and offers optional extras for multimodal models, distributed reward‑model inference, and more.

  • Dispatch
    Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate

    Hugging Face demonstrates sub‑millisecond per‑token generation for the 176B‑parameter BLOOM model using DeepSpeed‑Inference tensor parallelism and Accelerate pipeline parallelism on 8×80 GB A100 GPUs.

  • Project
    zai-org/GLM-5

    GLM‑5 is an open‑source series of very large LLMs (320 B‑744 B total parameters) optimized for multimodal input, 1 M‑token contexts, and long‑horizon agentic tasks. It introduces hybrid sparse/linear attention, IndexShare, and a custom RL engine (slime) to achieve frontier‑level performance on coding and planning benchmarks while offering cheap FP8 inference. Models are downloadable from Hugging Face/ModelScope and can be served with SGLang, vLLM, KTransformers, etc., with optional ‘reasoning_effort’ control.

  • Dispatch
    Open-Source Text Generation & LLM Ecosystem at Hugging Face

    Hugging Face’s 2023 blog post surveys the open‑source text generation and LLM ecosystem, covering model backgrounds, licensing, serving tools, and parameter‑efficient fine‑tuning.