Apple Faster Retrieval: Learned SupportNet and KeyNet for Amortized MIPS in RAG
Takeaway
Apple’s Faster Retrieval research demonstrates that neural routers—SupportNet for cluster‑wise support‑function scoring and KeyNet for query rewriting—outperform traditional centroid‑based routing in maximum inner product search (MIPS), achieving up to 8× lower latency for the same recall on large‑scale RAG benchmarks.
Problem Context: MIPS Bottlenecks in RAG
Maximum inner product search (MIPS) finds the key vector that maximizes the inner product with a query vector. In retrieval‑augmented generation (RAG), the best‑matching document is prefixed to the prompt. When the key database contains millions of vectors of thousands of dimensions, exhaustive search becomes prohibitively expensive.
Traditional approximate MIPS methods rely on geometric indexing (e.g., IVF clustering). They assume the query distribution resembles the key distribution, an assumption that fails for RAG where queries are short user questions and keys are long curated passages. Consequently, centroid‑based routing can mis‑route queries, missing the true nearest neighbor.
Core Insight: Support Functions as a Learned Router
The authors observe that the mapping from a query (q) to the maximal inner product (\max_{k}, q\cdot k) is the support function of the key set, a convex object in convex analysis. Two properties are crucial:
- Convexity – the support function can be approximated by Input Convex Neural Networks (ICNNs).
- Gradient = Best Key – the gradient of the support function at (q) yields the optimal key vector.
By training neural networks to regress either the support function values (SupportNet) or the optimal key directly (KeyNet), the system amortizes the cost of exhaustive search into a one‑time offline training phase.
Model Families
SupportNet
Learns the support function for each cluster.
- Architecture: loosely constrained ICNN.
- Output: a scalar score per cluster representing the best possible inner product within that cluster.
- Use case: high‑quality routing when a calibrated score is needed (e.g., re‑ranking or thresholding).
KeyNet
Learns to map a query directly to its best‑matching key vector.
- Architecture: standard feed‑forward network with a consistency loss derived from Euler’s theorem for homogeneous functions.
- Output: a transformed query vector that is already close to the target key.
- Use case: when inference speed dominates and a raw key approximation suffices.
Both models are trained on a supervised dataset generated by sampling queries from the target distribution, performing exhaustive MIPS to obtain ground‑truth scores or keys, and regressing to those targets.
Experimental Setup
- Dataset: Natural Questions (≈2.6 M passages) split into 128 clusters.
- Model sizes: XS, S, M, L corresponding to ~1 %, 5 %, 10 %, 20 % of the database size.
- Baseline: centroid routing in an IVF index (FAISS implementation).
- Metrics: routing recall (fraction of queries whose true cluster is among those probed) and overall retrieval recall as a function of latency or fraction of database scanned.
Results: SupportNet Routing
| Model | 95 % Routing Recall Latency | 97 % Routing Recall Latency |
|---|---|---|
| Centroid (baseline) | 3.36 ms | – |
| SupportNet XS | 1.74 ms | – |
| SupportNet L | 1.74 ms (97 % recall) | – |
- Latency‑adjusted recall: SupportNet achieves the same recall as the centroid router up to 1.9× faster.
- Quality‑adjusted recall: Scanning only 5 % of clusters, SupportNet recovers the true partition 90–93 % of the time, versus 87.5 % for centroids.
These gains hold across all model sizes; larger models further close the gap to perfect routing.
Results: KeyNet Query Rewriting
| Model | 99 % Recall Latency | Fraction of Database Scanned for 99 % Recall |
|---|---|---|
| Raw query (FAISS IVF) | 1.8 ms | 0.10 % |
| KeyNet L | 0.23 ms | 0.009 % |
- Latency reduction: The largest KeyNet reduces time to 99 % recall by ~8×.
- Search‑budget efficiency: With only 0.001 % of the database scanned, KeyNet achieves 89–97 % recall, outperforming raw queries (86 %).
Qualitative examples on Natural Questions show that KeyNet‑mapped queries surface more relevant top‑3 hits compared to raw queries.
Practical Guidance
- When to use KeyNet: In latency‑critical pipelines where the query‑to‑key mapping can be approximated and the downstream index is already highly optimized.
- When to use SupportNet: When a calibrated similarity score is required (e.g., for dynamic re‑ranking) or when the model budget is extremely tight; small SupportNets still beat centroids.
- Hybrid deployment: For massive key sets, a two‑stage system can be built—SupportNet routes to a subset of clusters, then a per‑cluster KeyNet refines the query before the standard IVF search.
Both models integrate with existing approximate‑search libraries (FAISS IVF, etc.) and are robust to modest distribution shifts in query patterns.
Community Reactions (Hacker News)
"Spotify’s Parallel Fusion Router uses a teacher LLM to train a tiny router LLM. Learned methods excel at hyperscale (hundreds of millions of items) but for a few million items BM25 + LLM prompting still wins." – armcat
"Retrieval heuristics can bias LLMs; letting the model explore without a hard router sometimes yields higher final quality." – comandillos
"Apple’s slow‑burn approach may eventually power on‑device personal‑assistant AI, leveraging local inference for privacy and cost advantages." – lavendermon
These comments highlight that while learned routers shine at scale, traditional lexical methods remain competitive for smaller corpora, and that bias introduced by hard routing must be weighed against speed gains.
Conclusion
Apple’s Faster Retrieval paper introduces a principled, convex‑analysis‑driven framework for amortized MIPS. By training tiny SupportNet and KeyNet models on the actual query distribution, the approach replaces centroid routing with learned routers that deliver substantially lower latency for the same recall, or higher recall for a fixed latency budget. The work bridges theoretical convex geometry with practical RAG systems and opens a path toward query‑aware, learned retrieval pipelines that can be dropped into existing IVF‑style indexes.