thu-nics/C2C

Cache‑to‑Cache (C2C) – Direct Semantic Communication Between Large Language Models

What it is – C2C is a Python library that lets two (or more) LLMs exchange information by translating and fusing their KV‑caches (the key‑value memory used by transformer attention) instead of sending text back‑and‑forth. By operating on the hidden‑state representations directly, the system can improve answer quality (≈ 8–10 % higher accuracy) and cut inference latency by about 2×.

Why the name “Rosetta” – Just as the Rosetta Stone let scholars read Egyptian hieroglyphs, the C2C “Rosetta” package learns a common “language” for KV‑cache tensors so that otherwise independent models can understand each other.

---

Key Features

Feature	What it does
KV‑Cache projection & fusion	Learns lightweight projector networks that map the cache of a sharer model into the semantic space of a receiver model.
Pre‑trained “Fusers”	Ready‑to‑use projector checkpoints for several popular model pairs (e.g., Qwen3‑0.6B ↔ Qwen2.5‑0.5B, Llama‑3.2‑1B, etc.) hosted on Hugging Face.
Multi‑sharer support	Fuse caches from several teacher models into a single receiver (still experimental).
Training‑only projector	During fine‑tuning only the projector weights are updated; the source and target LLMs stay frozen, making training cheap.
Simple API	A RosettaModel wrapper works like a normal transformers model; you just pass an extra kv_cache_index tensor to tell the system when to apply a projection.
Demo & scripts	Gradio demo, live‑chat example, and full training/evaluation scripts are included.
Future roadmap	Planned “agent‑managed KV‑Cache” serving system (2026‑09).

---

Typical Use‑Cases

• Ensemble reasoning – combine the latent knowledge of two (or more) LLMs without generating intermediate text, useful for complex QA or multi‑step reasoning.
• Speed‑critical inference – halve the latency compared to a naïve generate‑then‑communicate pipeline.
• Cross‑architecture knowledge transfer – let a newer, smaller model benefit from the hidden‑state semantics of a larger teacher.
• Research on LLM internals – study how KV‑cache representations encode information and how they can be transformed.

---

Quick Start (Installation & Running)

# 1. Create a fresh conda env (Python 3.10)
conda create -n rosetta python=3.10 && conda activate rosetta

# 2. Install the package (editable mode)
pip install -e .
# optional extras for training/evaluation
pip install -e ".[training,evaluation]"

Running the pre‑trained demo

import torch
from huggingface_hub import snapshot_download
from script.playground.inference_example import load_rosetta_model, run_inference_example

ckpt = snapshot_download(
    repo_id="nics-efc/C2C_Fuser",
    allow_patterns=["qwen3_0.6b+qwen2.5_0.5b_Fuser/*"],
)

cfg = {
    "rosetta_config": {
        "base_model": "Qwen/Qwen3-0.6B",
        "teacher_model": "Qwen/Qwen2.5-0.5B-Instruct",
        "checkpoints_dir": f"{ckpt}/qwen3_0.6b+qwen2.5_0.5b_Fuser/final",
    }
}

model, tokenizer = load_rosetta_model(cfg, eval_config={}, device=torch.device("cuda"))
prompt = [{"role": "user", "content": "Say hello in one short sentence."}]
input_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

# tell the model to apply the projector on the first token
instr_idx = torch.tensor([1, 0], dtype=torch.long).repeat(inputs['input_ids'].shape[1]-1, 1).unsqueeze(0).to(model.device)
label_idx = torch.tensor([-1, 0], dtype=torch.long).unsqueeze(0).to(model.device)
kv_idx = [instr_idx, label_idx]

with torch.no_grad():
    out = model.generate(**inputs, kv_cache_index=kv_idx, do_sample=False, max_new_tokens=64)
    print(tokenizer.decode(out[0], skip_special_tokens=True))

The script prints the C2C‑generated answer.

Interactive chat example

# single sharer
python script/playground/live_chat_example.py --checkpoint_dir path/to/checkpoint

# multiple sharers (list all checkpoint dirs)
python script/playground/live_chat_example.py --checkpoint_dir ckpt1 ckpt2

A simple terminal chat will appear, showing how the receiver model’s responses are enriched by the sharer’s cache.

---

Training Your Own Projector

1. Create a config in recipe/train_recipe/ (see C2C_0.6+0.5.json).
2. Run:

   python script/train/SFT_train.py --config recipe/train_recipe/C2C_0.6+0.5.json   # single‑GPU
   # or multi‑GPU
   torchrun --nproc_per_node=8 script/train/SFT_train.py \
       --config recipe/train_recipe/C2C_0.6+0.5.json
   

   Only the projector layers are updated; the two LLMs stay frozen.

---

Evaluation

Prepare an eval yaml (e.g., recipe/eval_recipe/unified_eval.yaml) and run:

python script/evaluation/unified_evaluator.py --config recipe/eval_recipe/unified_eval.yaml

The evaluator loads the Rosetta model, runs generation on the chosen benchmark, and writes standard metrics (accuracy, latency, etc.).

---

Extending the Framework

• Add a new projector – implement a subclass of Projector in rosetta/model/projector.py and register it with @register_model.
• Add a new dataset – create a DatasetConfig in rosetta/train/dataset_adapters.py and reference it in the training JSON.
• Add a new benchmark – follow the pattern in script/evaluation/unified_evaluator.py.

---

Supported Model Pairs (pre‑trained)

Receiver	Sharer	Checkpoint
Qwen3‑0.6B	Qwen2.5‑0.5B‑Instruct	link
Qwen3‑0.6B	Llama‑3.2‑1B‑Instruct	link
…	…	…
(Full table in README.)

---

Citation

If you use C2C in research, cite the arXiv paper:

@article{fu2025c2c,
  title={Cache-to-Cache: Direct Semantic Communication Between Large Language Models},
  author={Tianyu Fu and Zihan Min and Hanling Zhang and Jichao Yan and Guohao Dai and Wanli Ouyang and Yu Wang},
  journal={arXiv preprint arXiv:2510.03215},
  year={2025}
}

---

Where to Find More

• Project page: https://fuvty.github.io/C2C_Project_Page/
• Paper: https://arxiv.org/abs/2510.03215
• Hugging Face model hub: https://huggingface.co/nics-efc/C2C_Fuser
• Live demo (Gradio): https://huggingface.co/spaces/nics-efc/C2C_demo

---

Related Work from the Same Group

• R2R – token‑level routing for reasoning LLMs
• TaH – selective latent thinking for reasoning LLMs
• FrameFusion – video token reduction for LVLMs
• MoA – mixture of sparse attention for LLMs

---

Bottom line: C2C (Rosetta) provides a practical, open‑source way to let LLMs talk to each other through their internal attention caches, delivering better accuracy and faster inference for ensemble‑style or teacher‑student scenarios.