gokayfem/ComfyUI_VLM_nodes

ComfyUI nodes for vision-language models: Qwen3-VL, Moondream 3, Florence-2, SmolVLM2, InternVL, Gemma 3, MiniCPM-V. Plus open-vocabulary detection, SAM2/SAM3 segmentation, video temporal reasoning, GGUF via llama.cpp, and hosted LLM/VLM APIs.

What it solves

This project provides a comprehensive suite of production-oriented nodes for ComfyUI to integrate Vision-Language Models (VLMs), structured vision tasks, and advanced text processing. It bridges the gap between raw VLM outputs and usable data by providing tools for structured detection, segmentation, tracking, and adaptive video analysis, while optimizing for VRAM and inference speed.

How it works

The toolkit implements a series of specialized nodes that handle the entire VLM pipeline:

  • Model Integration: A curated "Modern VLM" interface supports a wide range of models (e.g., Qwen, SmolVLM2, Gemma 3) using native Transformers implementations.
  • Vision Pipeline: It uses a structured data flow (sockets like VLM_DETECTIONS and VLM_TRACKS) to pass spatial data between nodes without relying on model-specific formats.
  • Adaptive Video Intelligence: To avoid expensive per-frame inference, it employs adaptive frame sampling based on motion and scene changes, and a temporal reasoner to summarize video content.
  • VRAM Optimization: It includes performance utilities to downsize image pixel budgets and manage model residency/offloading to prevent memory overflows.
  • Text Toolkit: A set of utility nodes for cleaning, splitting, joining, and parsing JSON from VLM responses into usable strings.

Who it’s for

  • ComfyUI Users: Those looking to integrate advanced image and video understanding into their generative workflows.
  • AI Developers: Users needing structured vision outputs (bounding boxes, masks, polygons) for downstream tasks.
  • Robotics/Video Analysts: Developers building automated video reasoning or tracking pipelines.

Highlights

  • Broad Model Support: Integrated support for Qwen, SmolVLM2, InternVL, Gemma 3, and specialized models like Florence-2 and Moondream.
  • Live Streaming: Decoded text streams in real-time through ComfyUI's WebSocket channel.
  • Structured Spatial Data: Standardized schemas for detections, tracks, and events, making it easy to convert VLM outputs to masks or crops.
  • Efficient Video Processing: Adaptive sampling and pixel-budgeting that can significantly reduce the analysis workload before inference.
  • Advanced Segmentation: Integration with SAM2.1 and an adapter for ComfyUI core SAM3.1 for high-quality object tracking and masking.

Related

  • Project
    heshengtao/comfyui_LLM_party

    A comprehensive set of ComfyUI nodes for building LLM workflows, enabling the integration of AI agents, RAG, and VLMs into visual image generation pipelines.

  • Project
    1038lab/ComfyUI-QwenVL

    A ComfyUI custom node suite that integrates Qwen-VL vision-language models for image and video analysis, featuring support for both Transformers and GGUF backends.

  • Dispatch
    Hugging Face Vision Language Models 2025 Update

    Hugging Face provides a comprehensive overview of the 2024-2025 evolution of Vision Language Models (VLMs), highlighting trends in any-to-any architectures, reasoning models, and the rise of Vision-Language-Action (VLA) models for robotics.

  • Dispatch
    nanoVLM: A Minimalist PyTorch Toolkit for Training Vision Language Models

    Hugging Face has released nanoVLM, a lightweight, pure PyTorch toolkit designed to simplify the training and understanding of Vision Language Models (VLMs) for beginners and researchers.

  • Project
    Acly/comfyui-tooling-nodes

    A set of ComfyUI custom nodes and API extensions that enable the use of ComfyUI as a backend for external tools, featuring optimized image transfer, regional prompting, and model inspection.