Run SmolVLM on Intel CPUs with OpenVINO in 3 Steps

TL;DR

Hugging Face demonstrates that the SmolVLM vision‑language model can be deployed on any Intel CPU in three simple steps—conversion to OpenVINO IR, optional INT8 quantization, and inference—delivering up to a 12× reduction in time‑to‑first‑token and a 65× increase in decoding throughput compared with the original PyTorch version.

1. Deploy the Model with Optimum‑Intel

Takeaway: Installing optimum-intel[openvino] and transformers provides a unified CLI and Python API that automatically handles OpenVINO export and runtime execution.

pip install optimum-intel[openvino] transformers==4.52.*

The optimum package abstracts away the low‑level OpenVINO details, letting you focus on model conversion and quantization.

2. Step‑by‑Step Guide

2.1 Convert to OpenVINO IR

Takeaway: The model must be converted to OpenVINO’s Intermediate Representation (IR) before it can be quantized or run on Intel hardware.

  • CLI conversion
    optimum-cli export openvino -m HuggingFaceTB/SmolVLM2-256M-Video-Instruct smolvlm_ov/
  • Python on‑the‑fly conversion
    from optimum.intel import OVModelForVisualCausalLM
model_id = "HuggingFaceTB/SmolVLM2-256M-Video-Instruct"
model = OVModelForVisualCausalLM.from_pretrained(model_id)
model.save_pretrained("smolvlm_ov")

Both methods produce the same OpenVINO IR files (.xml and .bin).

2.2 Quantization Options

Takeaway: Post‑training quantization reduces model size and latency; weight‑only quantization (WOQ) is a low‑risk first step, while static quantization further speeds up the vision encoder at the cost of a small accuracy trade‑off.

Option 1 – Weight‑Only Quantization (WOQ)

  • Quantizes only the weights to INT8, leaving activations in FP32.
  • Minimal accuracy impact; modest speed gains.
  • Since OpenVINO 2024.3, runtime activation quantization is automatically applied when weights are INT8, yielding additional speed.
from optimum.intel import OVModelForVisualCausalLM, OVWeightQuantizationConfig
q_config = OVWeightQuantizationConfig(bits=8)
q_model = OVModelForVisualCausalLM.from_pretrained(model_id, quantization_config=q_config)
q_model.save_pretrained("smolvlm_int8")

Or via CLI:

optimum-cli export openvino -m HuggingFaceTB/SmolVLM2-256M-Video-Instruct \
    --weight-format int8 smolvlm_int8/

Option 2 – Static Quantization (Mixed)

  • Quantizes both weights and activations after a calibration run on a representative dataset (e.g., 50 samples from the contextual dataset).
  • Applies static INT8 to the vision encoder while keeping the language model weights at WOQ precision.
  • Provides the greatest latency reduction for multi‑image or short‑answer scenarios.
from optimum.intel import (
    OVModelForVisualCausalLM,
    OVPipelineQuantizationConfig,
    OVQuantizationConfig,
    OVWeightQuantizationConfig,
)
q_config = OVPipelineQuantizationConfig(
    quantization_configs={
        "lm_model": OVWeightQuantizationConfig(bits=8),
        "text_embeddings_model": OVWeightQuantizationConfig(bits=8),
        "vision_embeddings_model": OVQuantizationConfig(bits=8),
    },
    dataset=dataset,
    num_samples=num_samples,
)
q_model = OVModelForVisualCausalLM.from_pretrained(model_id, quantization_config=q_config)
q_model.save_pretrained("smolvlm_static_int8")

2.3 Run Inference

Takeaway: After conversion (and optional quantization), inference is a single generate call; adding device="gpu" enables Intel GPU acceleration when available.

generated_ids = q_model.generate(**inputs, max_new_tokens=100)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)
print(generated_texts[0])

For GPU:

model = OVModelForVisualCausalLM.from_pretrained(model_id, device="gpu")

A public Hugging Face Space lets you experiment with the original, WOQ, and mixed‑quantized models on a 4th‑Gen Intel Xeon (Sapphire Rapids) system.

3. Performance Evaluation

Takeaway: OpenVINO conversion alone cuts the time‑to‑first‑token (TTFT) from 5.15 s to 0.42 s (≈12× faster) and raises decoding throughput from 0.72 tokens / s to 47 tokens / s (≈65×). Adding 8‑bit WOQ further reduces TTFT to 0.247 s and boosts throughput to 63.9 tokens / s.

Configuration TTFT (s) TPOT (s) End‑to‑End Latency (s) Throughput (tokens/s)
PyTorch 5.150 1.385 25.927 0.722
OpenVINO 0.420 0.021 0.738 47.237
OpenVINO 8‑bit WOQ 0.247 0.016 0.482 63.928

The benchmark used a single image input on an Intel Core Ultra 7 265K (20 threads, 64 GB DDR5) running Ubuntu 24.10 with OpenVINO 2025.2.0.

Platform configuration (excerpt): Intel Core Ultra 7 265K, 20 threads, 64 GB DDR5 @ 6400 MHz, OpenVINO 2025.2.0, optimum‑intel 1.25.2, transformers 4.53.3.

4. Resources

Disclaimer: Performance varies by configuration and workload. Results reflect the hardware and software versions listed above and may not represent all future updates.

Sources