AIFrontierLab/TorchUMM

A unified multimodal model toolkit

What it solves

TorchUMM provides a unified interface for the inference, evaluation, and post-training of multimodal models. It eliminates the need to manage separate codebases for different models that perform image understanding (Image $\rightarrow$ Text), image generation (Text $\rightarrow$ Image), and image editing, allowing users to run and compare them through a shared, config-driven framework.

How it works

The toolkit uses a pluggable model architecture with adapters for various multimodal backbones. It abstracts the complexity of model-specific requirements into YAML configurations, enabling users to switch between models and benchmarks without changing code. The framework supports local execution via a CLI, cloud-native scaling via Modal, and specialized execution for AMD ROCm clusters. It also includes a dedicated pipeline for batch inference and a Python API for programmatic access.

Who it’s for

Researchers and developers working with multimodal AI who need to evaluate multiple models across different benchmarks or perform post-training (such as SFT or UniCot) using a consistent set of tools.

Highlights

  • Unified Interface: Supports a wide array of unified multimodal models (e.g., Bagel, Emu3.5, Janus-Pro) and single-task models (e.g., FLUX.2, Qwen2.5-VL).
  • Comprehensive Evaluation: Integrated support for 10+ benchmarks covering generation, understanding, and editing (e.g., MME, MMBench, GenEval, WISE).
  • Post-Training Toolkit: Includes methods for SFT, IRG, recA, and UniCot.
  • Cloud-Native Scaling: Seamless integration with Modal for GPU scaling and automated output uploads to HuggingFace.
  • vLLM Integration: Native vLLM integration for specific models like Emu3.5 to optimize inference speed.

Related

  • Project
    meta-pytorch/torchtune

    torchtune is a PyTorch‑based library that provides ready‑to‑run recipes and configs for fine‑tuning, RL‑HF, distillation, quantization‑aware training, and inference of many modern LLMs (Llama, Gemma, Qwen, Phi, etc.). It abstracts distributed training, memory‑saving tricks, and device support behind a simple CLI (`tune`) and YAML files, integrating with Hugging Face, torch‑ao, FSDP2, and logging tools. Though development stopped in 2025, it remains a practical, open‑source toolkit for LLM experimentation.

  • Project
    EvolvingLMMs-Lab/lmms-eval

    A unified evaluation toolkit for Large Multimodal Models that provides reproducible and efficient benchmarking across image, video, and audio tasks.

  • Project
    Ma-Zhuang/OmniNWM

    OmniNWM is a research codebase that builds a panoramic, multi‑modal world model for autonomous‑driving simulation. It jointly generates RGB, semantic, depth, and 3‑D occupancy maps from a vehicle trajectory, uses normalized Plücker ray‑maps for precise control, and provides occupancy‑based dense rewards for closed‑loop policy evaluation. The repo includes installation steps (including a manual patch to `transformers`), data preparation instructions for nuScenes, pretrained checkpoint download links, and scripts for inference, out‑of‑distribution testing, and staged training.

  • Project
    jmaczan/tiny-vllm

    tiny‑vllm is an educational project that guides you through building a C++/CUDA LLM inference engine (a smaller, learning‑focused sibling of vLLM). It includes full source code and a step‑by‑step tutorial covering model loading, KV‑cache, static/continuous batching, FlashAttention‑style online softmax, and PagedAttention, with support for both NVIDIA (CUDA) and AMD (HIP) GPUs.

  • Project
    pytorch/torchtitan

    torchtitan is a PyTorch‑native library for large‑scale LLM pre‑training, fine‑tuning, and reinforcement‑learning, offering composable parallelism (FSDP, TP, PP, CP, EP), recipe‑based configuration, and support for frontier models and low‑precision formats.