1038lab/ComfyUI-QwenVL

What it solves

This project provides a set of custom nodes for ComfyUI that integrate the Qwen-VL series of vision-language models (LVLMs) from Alibaba Cloud. It allows users to perform multimodal AI tasks—such as image understanding, video analysis, and text generation—directly within their ComfyUI workflows without needing separate external tools.

How it works

The integration supports two primary backends:

• Transformers (HF): Uses the Hugging Face Transformers library for standard model loading and inference.
• GGUF (llama.cpp): Uses llama-cpp-python to run quantized GGUF models, which is often more memory-efficient for users with limited hardware.

It features an intelligent video auto-scaling system that calculates a token budget per frame to prevent CUDA Out-of-Memory (OOM) errors and context overflow. It also supports various attention mechanisms (SageAttention, Flash Attention 2, and SDPA) to optimize performance based on the user's GPU architecture.

Who it’s for

ComfyUI users who want to add vision-language capabilities to their generative AI workflows, ranging from those seeking simple "plug-and-play" nodes to advanced users who need fine-grained control over sampling, quantization, and memory management.

Highlights

• Multimodal Support: Handles both single images and video frame sequences as input.
• Flexible Quantization: Offers on-the-fly 4-bit, 8-bit, and FP16 options to balance VRAM usage and quality.
• Hardware Optimization: Includes SageAttention support with kernels optimized for Ampere, Ada, Hopper, and Blackwell GPUs.
• Integrated Model Management: Features an automatic downloader for Hugging Face models and a unified configuration system for custom models.
• Prompt Enhancers: Includes dedicated nodes for text-only prompt optimization using Qwen3 models.