Reame CPU Inference Server – Faster Over Time with Persistent KV Cache

Reame speeds up as it runs by persisting KV cache on disk

Reame is a CPU‑only inference server that becomes faster the longer it is used because it stores the transformer’s key‑value (KV) cache on persistent storage. This design lets subsequent requests reuse previously computed attention states, reducing the amount of work the CPU must repeat.

Persistent KV cache is the core performance trick

The server writes the KV cache to a file after each request and reloads it for future queries. By avoiding recomputation of earlier layers, the latency drops noticeably for workloads that share context (e.g., chat histories or repeated prompts). The approach is especially useful when running large language models on modest hardware such as the free Oracle Cloud ARM instances with 2 cores and 12 GB RAM that some users have highlighted.

CPU‑only deployment lowers cost and complexity

Because Reame does not require a GPU, it can run on inexpensive cloud VMs or on‑premise servers. Users can place model files in a models directory and configure the path in reame.conf. One commenter reported a "No such file or directory" error when attempting to load TinyLlama, indicating that proper placement of the model file and correct configuration are essential for successful startup.

Model selection is manual via configuration

Reame does not provide an automatic model picker; the user must edit reame.conf to point to the desired model file. This manual step gives full control over which model is served but also requires careful file management. A community member suggested that keeping models inside the Reame folder simplifies tracking, whereas placing them in /opt can lead to forgotten files.

Community observations and questions

"The persistent KV cache is interesting; I’d love to see how much of the speedup remains when requests share less structure." – ohadkr

This comment highlights a key research question: the cache‑based speedup depends on overlap between successive prompts. If later requests diverge significantly, the benefit may diminish.

"Why qwen 2.5 everywhere? Why not 3.5?" – tyzoid

The repository currently ships examples using the Qwen‑2.5 model, likely because it balances size and performance for CPU inference. Users can replace it with any compatible model, such as Qwen‑3.5, provided the model fits the memory constraints.

"Sick that you can get 2 arm cores and 12 GB ram for free at Oracle cloud, did not know that" – K0IN

Free tier cloud instances make Reame’s low‑cost, CPU‑only approach practical for experimentation and small‑scale deployments.

Documentation appears AI‑generated

One commenter noted that the repository and its README look AI‑generated, suggesting that the documentation may lack depth or contain inaccuracies. Users should verify configuration steps and test the server in a controlled environment before relying on it for production workloads.

Getting started with Reame

1. Clone the repository: git clone https://github.com/swellweb/reame.
2. Place a compatible model file (e.g., TinyLlama, Qwen‑2.5) in the models directory.
3. Edit reame.conf to set model_path = ./models/<model_file>.
4. Run the server: ./reame.
5. Send inference requests to the exposed HTTP endpoint.

Limitations and open questions

• Cache effectiveness: The speedup is proportional to prompt similarity; workloads with highly variable inputs may see limited gains.
• Disk I/O overhead: Persisting large KV caches can introduce read/write latency, especially on slower storage.
• Memory footprint: Storing KV states for long contexts can consume significant RAM, potentially exceeding the modest resources of free cloud tiers.

Conclusion

Reame demonstrates that CPU‑only inference can be made more efficient by persisting transformer KV caches, turning idle storage into a performance accelerator. While the approach shines for repetitive or context‑rich workloads on low‑cost hardware, its benefits depend on prompt overlap and careful resource management.