Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint

TL;DR: IBM's Granite Switch 4.1 release packages chat, general and specialized decision-making, and speech-to-text into single 8B and 30B model endpoints. The release matters because it adds these capabilities without increasing GPU memory beyond the base model, and it makes the fastest ASR model on the OpenASR Leaderboard available alongside the base LLM.

What Granite Switch 4.1 includes

Granite Switch 4.1 is a modular model composition approach introduced by the Granite Switch project. The 4.1 release provides Granite-Switch-4.1-8B and Granite-Switch-4.1-30B models configured with three additions: a new Decision aLoRA for general-purpose decision-making (e.g. v1/systemone), a specialized Guardian aLoRA for safety-related decision tasks, and Granite-Speech-5.0-Turbo-CTC for transcription. All three technologies and the underlying base model are composed into one model.

Decision and Guardian adapters

The Decision aLoRA handles general-purpose decision-making. The Guardian aLoRA is specialized for safety-related decision tasks and, on average, outperforms Jev 1.13, Microsoft-Decision-1, GPT-Luna-Decisions, and Clef-Flash on those tasks. The post positions these adapters as part of a broader decision-model strategy: Granite Libraries already provides a repository of specialized decision model adapters and other specialized task adapters that can be composed into custom Granite Switch models. While Jev and other closed decision models focus on generality, specialized granitelib adapters can outperform generalists on specialized classification tasks.

Speech-to-text with Granite-Speech-5.0-Turbo-CTC

Granite-Speech-5.0-Turbo-CTC is described as the fastest ASR model on the OpenASR Leaderboard. It is embedded in Granite Switch 4.1 for transcription tasks. A co-located ASR encoder helps the model switch between chat, decision, and spoken requests. The Speech-to-Decisions Hugging Face space demonstrates granite-switch-4.1-8b transcribing audio, generating summaries, screening for harmful content, and testing 15 other user-defined flags, all running on a single endpoint in real time.

Efficiency: one endpoint, base-model VRAM, KV cache reuse

Granite Switch 4.1 composes the base model and adapters into approximately the same amount of VRAM on a GPU as the base model would use by itself. That means more capabilities with fewer GPUs. The Switch adapters are built as cache-efficient Activated LoRAs, so the KV cache created from a base model generation can be reused for any subsequent adapter-based decisions. The release also incorporates recent vLLM contributions to speed up low rank adapter inferencing. Because adapters share the base model's KV cache, adding specialists alongside a generalist decision adapter costs almost nothing. This gives broad coverage from the generalist and higher accuracy from specialists on tasks that matter.

Performance comparison: RAG workflow and Doom

In a race between Granite-Switch-4.1-30B (frozen base Granite plus 4 specialized aLoRAs) and Jev-27B (frozen base Qwen model plus 1 generic decision LoRA), Granite-Switch-4.1-30B completed a real-world, 5-step RAG workflow much faster than its competitor. In a separate demonstration, a custom 3B parameter Granite Switch model made split-second decisions to navigate the classic game Doom at 60-100 decisions per second, compared to Jev's reported 10 decisions per second. Unlike Jev, this Granite Switch model can also narrate and respond to questions while it is playing.

Compose your own Granite Switch model

Granite-Switch-4.1-8B and -30B are released with 'batteries included' as predefined sets of helpful adapters. Users do not need to use that exact composition. Anyone can compose their own Granite Switch model by mixing and matching adapters from Granite Libraries. The Granite team plans to release an Adapter SDK for training adapters in Mellea, alongside other tutorials and Granite Switch support. These adapters could be specialized classification adapters to augment decision-based workflows or adapters for specialized language tasks. The post notes that custom adapters could even teach a 3B parameter version of Granite to play Doom. More details and data on Granite Switch are planned for release in the upcoming weeks.

Resources for getting started

Sources