mattmireles/gemma-tuner-multimodal

Fine-tune Gemma 4 and 3n with audio, images and text on Apple Silicon, using PyTorch and Metal Performance Shaders.

Gemma Multimodal Fine‑Tuner (macOS‑only)

What it is – A Python package that lets you fine‑tune Google’s Gemma language models (both the 3‑nano and 4‑series) with LoRA adapters on text, images, and audio data, all from a Mac with Apple‑Silicon (MPS) – no NVIDIA GPU required. The tool includes a small CLI wizard, a real‑time web visualiser, and built‑in support for streaming huge datasets from Google Cloud Storage or BigQuery.

Core capabilities (as described in the README)

Feature Details
Modality support • Text‑only (instruction or completion)\n• Image + text (captioning or VQA)\n• Audio + text (ASR‑style)
Hardware Runs natively on Apple Silicon via PyTorch MPS; optional CUDA support if you have an NVIDIA GPU.
Data ingestion CSV files (train/validation) are the only required format. CSV rows can reference local files, HTTP URLs, GCS paths, or BigQuery tables; the prepare command streams data so you never need to copy terabytes locally.
Training UI A browser‑based visualiser (loss curve, attention heat‑map, gradient signal, memory usage, token predictions) starts automatically when visualize = true in a profile.
Model targets Gemma‑3n (2 B & 4 B) and Gemma‑4 (2 B & 4 B) checkpoints from Hugging Face. LoRA adapters are merged into a single SafeTensors/HF tree for export.
Export gemma-macos-tuner export produces a merged model ready for Core ML conversion or GGUF inference.
CLI wizard Guided setup (gemma-macos-tuner wizard) creates a config file, picks a model, selects a dataset profile, and launches training with a single click.
Streaming & big data prepare can stream audio shards from GCS/BigQuery; the same pipeline works for image/audio modalities, letting you train on multi‑TB corpora without filling your SSD.
Package layout • gemma_tuner/cli_typer.py – entry‑point CLI (gemma-macos-tuner).\n• gemma_tuner/scripts/finetune.py – dispatches to the Gemma‑specific trainer.\n• gemma_tuner/models/gemma/finetune.py – PEFT LoRA fine‑tuning logic.\n• gemma_tuner/wizard/ – Rich/Questionary UI for the wizard.

Typical use‑cases (from the README)

  • Domain‑specific speech recognition – adapt Gemma + audio LoRA to medical dictation, legal transcripts, call‑center recordings, etc.
  • Specialised vision tasks – fine‑tune on receipts, charts, manufacturing defects, or medical images for captioning or VQA.
  • Document & UI understanding – train on screenshot → structured‑output pairs for on‑device assistants.
  • Low‑resource language or accent adaptation – improve transcription or translation for under‑represented dialects.
  • Private on‑device pipelines – all training and inference stay on the Mac; no data leaves the machine.

Getting started (summarised from the README)

  1. Create a native arm64 Python 3.10+ venv (Homebrew python@3.12 works).
  2. Install PyTorch (pip install torch torchaudio).
  3. Install the package (pip install -e .).
  4. Log in to Hugging Face and accept the Gemma model license.
  5. (Optional) Install Gemma‑4 extras (pip install -r requirements/requirements-gemma4.txt).
  6. Run the wizard – gemma-macos-tuner wizard. It will generate config/config.ini and walk you through model, dataset, and hyper‑parameter selection.
  7. Train – either via the wizard or directly: gemma-macos-tuner finetune <profile>. Use --json-logging for machine‑readable logs.
  8. Export – gemma-macos-tuner export <run‑dir-or‑profile> produces a merged checkpoint ready for Core ML or GGUF.

Limitations & known issues (as documented)

  • Image & audio streaming – only audio pipelines currently support GCS/BigQuery streaming; image fine‑tuning still requires local CSV files.
  • Gemma 4 large checkpoints (26 B/31 B) are not supported because the trainer’s AutoModelForCausalLM audio path expects a different architecture.
  • Audio‑only LoRA works on Apple Silicon; CUDA‑only paths are marked “⚠️” and may require an NVIDIA GPU.
  • Memory – Minimum 16 GB RAM; 32 GB+ recommended for the 4 B models. OOM can be mitigated by smaller batch sizes or gradient checkpointing.
  • MPS stability – The trainer forces eager attention and prefers bf16; fallback to CPU (PYTORCH_ENABLE_MPS_FALLBACK=1) is for debugging only and slows training.
  • Export for Gemma 4 – Some inference utilities still reject Gemma 4 IDs until those code paths are upgraded.

Quick reference cheat‑sheet (from the README)

# Prepare data (CSV → optional streaming)
 gemma-macos-tuner prepare <dataset‑profile>

# Train
 gemma-macos-tuner finetune <profile> --json-logging

# Evaluate
 gemma-macos-tuner evaluate <profile-or‑run>

# Export merged checkpoint
 gemma-macos-tuner export <run‑dir-or‑profile>

# Launch the wizard UI
 gemma-macos-tuner wizard

Bottom line – Gemma‑tuner‑multimodal is a purpose‑built, Apple‑Silicon‑first trainer that adds LoRA‑based multimodal fine‑tuning (text, image, audio) to the Gemma family, with a friendly CLI wizard and live visualisation, while allowing massive cloud‑resident datasets to be streamed directly to a Mac.

Related

  • Dispatch
    Gemma 4 release: open‑source multimodal models with on‑device support

    Gemma 4, a new open‑source multimodal family from Google DeepMind, is released on Hugging Face with image, audio, video support, up to 256 K context, and day‑0 compatibility with transformers, llama.cpp, MLX, Rust, and WebGPU.

  • Dispatch
    Fine-Tuning Gemma Models in Hugging Face

    Hugging Face provides a guide on using Parameter-Efficient Fine-Tuning (PEFT) and Low-Rank Adaptation (LoRA) to customize Google's Gemma models on GPUs and Cloud TPUs.

  • Dispatch
    Google Gemma Open LLM Release

    Google has released Gemma, a family of open-access large language models based on Gemini, available in 2B and 7B parameter sizes with base and instruction-tuned variants.

  • Project
    avifenesh/memra

    memra is a Rust + CUDA inference engine tuned for RTX PRO 6000 and RTX 5090 GPUs. It loads Hugging Face safetensors or GGUF checkpoints, serves them via an OpenAI‑compatible HTTP API, and offers features like prefix‑cache reuse, MTP speculative decoding, and image/video inputs. The project ships pre‑built binaries, provides a quick‑start script, and reports measured performance (e.g., 140 tok/s decode on a PRO 6000, 0.156 s TTFT). It supports a curated list of models (Qwen, Gemma‑4, Ornith, etc.) and can split very large models across two GPUs (PP‑2). Designed for single‑GPU or two‑GPU workstations, it does not aim for multi‑node tensor parallelism.

  • Dispatch
    Setting Up a Local Coding Agent on macOS with Gemma 4 and llama.cpp

    A technical guide to deploying a local coding agent on macOS using Gemma 4 26B, llama.cpp with Metal acceleration, and the Pi agent harness for an OpenAI-compatible offline workflow.