dwzhu-pku/PaperBanana

PaperBanana – AI‑powered academic illustration

What it does – PaperBanana is an open‑source, multi‑agent system that turns the text of a research paper (e.g., a methods paragraph and a figure caption) into a ready‑to‑publish diagram or plot. It does this by chaining together specialized agents:

1. Retriever – pulls similar reference figures from a curated dataset to give the system visual examples.
2. Planner – writes a detailed textual description of the desired figure based on the paper’s content.
3. Stylist – rewrites the description to follow academic style guidelines that the system automatically generates.
4. Visualizer – calls a state‑of‑the‑art image‑generation model (e.g., Gemini, OpenAI, Anthropic via OpenRouter) to create the picture.
5. Critic – evaluates the output, asks the Visualizer to improve it, and repeats for a few rounds.

The result is a set of candidate figures that can be inspected, refined, and downloaded at high resolution.

How to use it

• Web UI – a Gradio demo (also hosted on Hugging Face Spaces) or a Streamlit demo lets you paste text, pick a model, set options (aspect ratio, number of candidates, refinement rounds) and generate figures with a click.
• CLI – python main.py runs the full pipeline locally; flags let you choose dataset, task type (diagram vs. plot), retrieval mode, and experiment mode.
• Configuration – API keys for Gemini or OpenRouter are stored in a YAML file (configs/model_config.yaml). The system works with either provider; OpenRouter is preferred when both are present.

Key components

• Dataset – PaperBananaBench on Hugging Face supplies reference diagrams and ground‑truth plots for retrieval and evaluation.
• Modular agents – each agent lives in agents/ and can be swapped or extended.
• Evaluation tools – scripts under visualize/ let you view pipeline evolution and metric scores against the benchmark.
• Extensibility – you can add new style guides, plug in different image‑generation models, or enable manual example selection (TODO).

Why it matters – Creating clear, accurate figures is a bottleneck for researchers. PaperBanana demonstrates that large language models, combined with retrieval‑augmented prompting and iterative refinement, can automate much of this work, potentially speeding up the writing process and improving visual consistency across papers.

Community – The repo is a fork of Google‑Research’s PaperVizAgent and includes contributions from many developers. It also links to related community forks (e.g., Chinese‑enhanced versions) and other projects tackling automated figure generation.

Getting started

# clone and enter repo
git clone https://github.com/dwzhu-pku/PaperBanana.git && cd PaperBanana

# set up env with uv (or pip if you prefer)
uv venv && source .venv/bin/activate
uv python install 3.12
uv pip install -r requirements.txt

# copy config template and add your API key(s)
cp configs/model_config.template.yaml configs/model_config.yaml
# edit configs/model_config.yaml → fill in defaults.main_model_name, defaults.image_gen_model_name, and an api_keys entry

# download the benchmark data
# (place it under data/PaperBananaBench/ as instructed)

# run the web UI locally
python app.py   # Gradio
# or
streamlit run demo.py   # Streamlit

Bottom line – PaperBanana is a genuine AI research codebase that showcases a reference‑driven, multi‑agent pipeline for automatically generating academic figures, complete with a demo UI, CLI, and evaluation framework.