google-deepmind/reverb

Reverb – Scalable Experience‑Replay Service for Reinforcement Learning

What it is – Reverb is a Python‑first library (with a fast C++ core) that provides a networked data‑storage service for reinforcement‑learning (RL) research. It lets you run one or more servers that hold tables of items (e.g., transitions, trajectories, or any arbitrary tensors). Clients can insert data, sample it according to configurable strategies, and update priorities, all over gRPC. The design mirrors the experience replay buffers used by off‑policy RL algorithms, but the API is flexible enough for on‑policy, supervised, or even general data‑distribution workloads.

---

Core concepts (as described in the README)

Concept	What it does
Server	Holds one or more tables; each table defines capacity, sampling/removal policies, and rate‑limiting.
Table	A container for items; an item references one or more data elements (the actual tensors). Items are lightweight because they store only references, allowing many items to share the same data element.
Selectors	Define how items are chosen for sampling or removal. Built‑in selectors: Uniform, Prioritized, FIFO, LIFO, MinHeap, MaxHeap.
Rate limiters	Control when inserts or samples are allowed (e.g., MinSize, SampleToInsertRatio, Queue, Stack).
Sharding	Multiple independent Reverb servers can be run and load‑balanced; data is not replicated, so the system scales horizontally at the cost of possible data loss.
Checkpointing	Servers can dump their full state to disk and later be restored, enabling fault‑tolerance for long training runs.

---

Typical workflow (quick‑start excerpt)

import reverb
# 1️⃣ Define a server with a single table
server = reverb.Server(tables=[
    reverb.Table(
        name='my_table',
        sampler=reverb.selectors.Uniform(),   # how to pick items for sampling
        remover=reverb.selectors.Fifo(),      # how to drop items when full
        max_size=100,
        rate_limiter=reverb.rate_limiters.MinSize(1)),
])

# 2️⃣ Connect a client
client = reverb.Client(f'localhost:{server.port}')

# 3️⃣ Insert data (single element or a trajectory)
client.insert([0, 1], priorities={'my_table': 1.0})

# 4️⃣ Sample
samples = list(client.sample('my_table', num_samples=2))

The README also shows a richer trajectory writer that lets you build an item referencing a sequence of data elements before committing it.

---

Why use Reverb?

• Performance – Core is C++ with gRPC, and TensorFlow ops are provided for zero‑copy integration with tf.data pipelines.
• Flexibility – Choose any combination of selectors, rate limiters, and table sizes; support FIFO queues, prioritized replay, or custom heaps.
• Distributed training – Run many servers behind a load balancer; clients automatically spread inserts/samples across them.
• Persistence – Checkpoint and restore a whole replay buffer without losing priorities or ordering.
• Beyond RL – The README notes that users have stored model weights or other frequently‑updated data, using Reverb as a lightweight in‑memory distributed file system.

---

Installation & compatibility

• Install via pip: pip install dm-reverb[tensorflow] (adds the matching TensorFlow version) or pip install dm-reverb if you manage TF yourself.
• Nightly builds are available as dm-reverb-nightly.
• Linux‑only; the library is not hardened for production use and may segfault.
• Specific TensorFlow versions are tied to Reverb releases (e.g., 0.14.0 works with TF 2.14.0).

---

Limitations noted in the README

• Not production‑ready – Intended for research; stability guarantees are limited.
• Linux‑only – No Windows/macOS support out of the box.
• No built‑in replication – Scaling relies on sharding; data loss is acceptable in the intended use‑case.
• Version coupling – Must match the TensorFlow version bundled with the wheel; mismatched TF can cause install failures.

---

When to reach for Reverb

If you are building a reinforcement‑learning system that needs a fast, distributed replay buffer with features like prioritized sampling, custom removal policies, or checkpointable state, Reverb gives you a ready‑made service rather than hand‑rolling one. It is also handy for any research pipeline that benefits from a shared, in‑memory store for large tensors across multiple processes or machines.

---

Citation

If you use Reverb in a publication, cite the accompanying paper:

@misc{cassirer2021reverb,
  title={Reverb: A Framework For Experience Replay},
  author={Albin Cassirer and Gabriel Barth-Maron and Eugene Brevdo and Sabela Ramos and Toby Boyd and Thibault Sottiaux and Manuel Kroiss},
  year={2021},
  eprint={2102.04736},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}