Creating a Coding Assistant with StarCoder – Summary

Overview

  • Goal: Convert StarCoder (a 16 B‑parameter code‑generation model) into a chat‑style coding assistant – StarChat‑α.
  • Key steps:
    1. Add special chat tokens (<|system|>, <|assistant|>, <|user|>, ).
    2. Prepare a dialogue dataset by filtering the Open‑Assistant oasst1 dataset to English‑only conversations and adding the new tokens.
    3. Mask user labels so the loss is computed only on assistant outputs.
    4. Fine‑tune the model using DeepSpeed ZeRO‑3 (or alternatives like LoRA/FSDP) because the model is too large for a single GPU.

Tokenizer & Masking

special_tokens = {
    "additional_special_tokens": ["<|system|>", "<|assistant|>", "<|user|>", "\n"]
}
tokenizer.add_special_tokens(special_tokens)
  • Verifies <|assistant|> maps to a single token ID (49153).
  • mask_user_labels replaces user‑turn token IDs and everything until the next assistant token with -100 so they are ignored by the loss.

Training with DeepSpeed ZeRO‑3

git clone https://github.com/bigcode-project/starcoder.git
cd starcoder/chat
conda create -n starchat python=3.10 && conda activate starchat
# install pytorch (see pytorch.org) then:
pip install -r requirements.txt
huggingface-cli login
sudo apt-get install git-lfs

torchrun --nproc_per_node=8 train.py config.yaml \
    --deepspeed deepspeed_z3_config_bf16.json
  • Uses an 8‑GPU A100 (80 GB) setup; training finishes in ~45 min.
  • config.yaml defines dataset paths, model name, training hyper‑parameters, etc.

Sample Outputs (Coding Tasks)

Prompt Model Output
Bar plot – plot height & age from a dict Generates a complete pandas/seaborn script with two sub‑plots and displays the figure.
World map – colour Germany & Spain red Produces a geopandas script that loads a GeoJSON, isolates the two countries, and plots them with the requested colors.
Basketball stats – scatter plot of points vs rebounds Returns a seaborn scatter plot with annotations for each player and proper titles/labels.

Evaluation

  • Standard benchmarks (ARC, HellaSwag, MMLU, TruthfulQA) show modest gains after instruction‑tuning:
    Model ARC HellaSwag MMLU TruthfulQA
    StarCoderBase 0.30 0.46 0.33 0.40
    StarChat‑α 0.33 0.49 0.34 0.44
  • Human‑in‑the‑loop / AI‑in‑the‑loop: Used a curated set of 115 Python prompts (from a ChatGPT‑generated seed) and had GPT‑4 and ChatGPT rate the model’s responses on a 1‑8 scale. StarChat‑α beat the base model ~96 % of the time.

Limitations & Risks

  • Hallucinations and unsafe content are still possible (no RLHF or safety filtering).
  • Demographic bias reflects the GitHub community from which most training data originates.
  • Benchmarks do not capture conversational quality; human or LLM‑based evaluation is needed.

Future Directions

  • Explore LoRA or other PEFT methods for cheaper fine‑tuning.
  • Combine code‑centric data with more natural‑language dialogue to improve conversational ability.
  • Deploy as an open‑source assistant for broader community use.

Resources

Citation

@article{Tunstall2023starchat-alpha,
  author = {Tunstall, Lewis and Lambert, Nathan and Rajani, Nazneen and Beeching, Edward and Le Scao, Teven and von Werra, Leandro and Han, Sheon and Schmid, Philipp and Rush, Alexander},
  title = {Creating a Coding Assistant with StarCoder},
  journal = {Hugging Face Blog},
  year = {2023},
  note = {https://huggingface.co/blog/starchat-alpha},
}

Sources

Related