EvoVLM – Evolving Multimodal Inference Programs

What it is

EvoVLM is a research‑grade code base that explores how to improve the inference behaviour of a frozen vision‑language model (the Qwen‑3‑VL‑2B family). Instead of fine‑tuning the model weights, it lets a large language model (LLM) propose small, typed mutations to the Python inference program – changing prompts, image preprocessing steps, generation budgets, or answer‑extraction logic. An evolutionary loop (mutation → evaluation → Pareto selection) keeps the best trade‑offs between accuracy and latency for a scientific chart‑understanding benchmark.

Why it matters

• Weight‑free optimisation – the model stays frozen, so the approach can be applied to any proprietary or large‑scale VLM without expensive retraining.
• Multi‑objective search – it explicitly balances correctness with speed, a practical concern for real‑time or edge deployments.
• Reproducible research – the repo ships the exact search traces, evaluation evidence, and a tiny “replay” mode that verifies results without needing the model files.

Core components (from the repo)

Component	Role
evolution/	Typed mutation definitions, the local LLM‑guided mutation engine, and the evolutionary search logic.
evaluation/	Scripts that run candidate programs on a small validation set, record accuracy, latency, errors, and provenance.
best_*.py	The final, selected inference programs that achieved the best accuracy‑latency Pareto points.
reproduce.sh & evaluation/portable_reproduce.py	CPU‑only “replay” commands that re‑run the recorded evidence to check that the published numbers are reproducible without downloading model weights.
report.pdf & generated/report_number_trace.json	The technical report and the machine‑readable trace of all metrics used in the paper.
data/ & charxiv/	Small subsets of the CharXiv chart dataset used for the search and for verification.

How to get started

1. Clone the repo and cd into research/evolving-vlm-inference-main.
2. Run the CPU‑only verification (no model download needed):

   python3 -B -m evaluation.portable_reproduce
   bash reproduce.sh check   # runs the recorded evidence checks
   bash reproduce.sh smoke   # imports the exported programs and validates their API
   

3. If you have a GPU and the Qwen‑3‑VL‑2B weights, follow the README.md inside the same folder to install the pinned requirements.lock.txt, download the model snapshots, and run the inference programs on your own data.

What the paper reports

Program	Correct / 128	Accuracy	Median latency (s)
Baseline Instruct	51	39.84%	14.54
Manual Instruct (human‑crafted)	78	60.94%	15.44
Evolved Instruct (from EvoVLM)	78	60.94%	14.07
Manual Thinking	23	17.97%	20.04
Evolved Thinking	44	34.38%	17.91

The evolved programs keep (or improve) the accuracy of their manual seeds while shaving 8–10 % off the median latency.

Limitations & scope

• The evolutionary search was tiny – only two generations with ≤ 6 children per population.
• Results are on the Dev128 validation split of the CharXiv chart benchmark; they are not a full leaderboard score.
• Latency numbers depend on the specific hardware/software environment used in the original runs.
• Model weights are external; the repo only contains the code and the recorded evidence.

---

TL;DR: EvoVLM shows a proof‑of‑concept that you can let an LLM “tweak” the inference script of a frozen vision‑language model, run a tiny evolutionary search, and obtain programs that are a bit faster without losing accuracy on a chart‑understanding task.