0814wdwd/solo_jev

What it solves

SOLO (System One Layout Optimizer) addresses the high latency and computational cost of running decision models on large datasets. Even when a model only needs to return a single token (a decision), it must process the entire input record (policies, context, user data). When records share repeated content, processing them in their original order often wastes computation by recalculating the same prefixes. SOLO optimizes the input layout by reorganizing rows and fields to maximize the reuse of prefix-caching backends (like vLLM), significantly increasing throughput and reducing prefill time.

How it works

SOLO operates as a backend-independent planner that reorganizes data before it is sent to an LLM-based decision model:

1. Normalization: It keeps all fields and original row identities.
2. Layout Planning: It analyzes the data to identify columns and row groupings that create the smallest distinct prefix combinations, maximizing the potential for cache hits.
3. Execution: It sends the reorganized records through a concurrent client to a prefix-cache-aware backend (such as vLLM).
4. Restoration: Once decisions are received, it restores the outputs to the original row order and index.

Who it’s for

This tool is designed for developers and researchers working with structured workloads where records contain repeated policies, account context, or correlated fields, and where prefill time is a significant portion of total execution time.

Highlights

• Significant Throughput Gains: Demonstrated up to 11.7x throughput increase over original order in synthetic workloads with long shared context.
• Backend Agnostic: The layout optimizer works independently of the specific inference engine, though it is optimized for prefix-caching backends.
• Flexible Input Support: Supports Pandas DataFrames, NumPy arrays, JSON/JSONL files, and nested objects.
• Integrated Decision Model: Includes a backend for AutoTrust JEV-9B, an open decision model built on Qwen3.5-9B.
• Diagnostic Tools: Includes an explain method to diagnose reusable structure in datasets via column NDVs and prefix combinations.