mattmireles/gemma-tuner-multimodal

Fine-tune Gemma 4 and 3n with audio, images and text on Apple Silicon, using PyTorch and Metal Performance Shaders.

What it solves

It provides a streamlined way to fine-tune Hugging Face Gemma models on multimodal data, including text, images, and audio, specifically optimized for Apple Silicon users.

How it works

The project uses Parameter-Efficient Fine-Tuning (PEFT) with Low-Rank Adaptation (LoRA) to make the training process more efficient. It includes a command-line wizard and a set of training tools to guide the user through the process.

Who it’s for

Developers and researchers using Apple Silicon (macOS) who want to customize Gemma models for multimodal tasks.

Highlights

  • Multimodal support for text, images, and audio.
  • Optimized for Apple Silicon hardware.
  • Uses PEFT LoRA for efficient fine-tuning.
  • Includes a command-line wizard for easier setup and execution.

Related

  • Dispatch
    Gemma 4 release: open‑source multimodal models with on‑device support

    Gemma 4, a new open‑source multimodal family from Google DeepMind, is released on Hugging Face with image, audio, video support, up to 256 K context, and day‑0 compatibility with transformers, llama.cpp, MLX, Rust, and WebGPU.

  • Project
    hiyouga/LlamaFactory

    LlamaFactory is an open‑source library that provides a zero‑code CLI and Gradio UI for fine‑tuning over 100 large language models (LLaMA, Qwen, Mistral, Mixtral‑MoE, etc.) using a wide range of techniques (LoRA, QLoRA, FlashAttention‑2, GaLore, etc.) and supports NVIDIA, AMD, and Ascend hardware. It includes built‑in experiment tracking, OpenAI‑style serving, and Docker images, and is used by Amazon, NVIDIA, and Alibaba.

  • Dispatch
    Fine-Tuning Gemma Models in Hugging Face

    Hugging Face provides a guide on using Parameter-Efficient Fine-Tuning (PEFT) and Low-Rank Adaptation (LoRA) to customize Google's Gemma models on GPUs and Cloud TPUs.

  • Project
    Blaizzy/mlx-vlm

    A package for inference and and fine-tuning of Vision Language Models and Omni Models on Mac using MLX, supporting multimodal inputs and speculative decoding for faster generation.

  • Project
    huggingface/huggingface-gemma-recipes

    A collection of minimal recipes and notebooks for implementing multimodal inference, fine-tuning, and RAG with the Gemma family of models.