Most production agent stacks wire three separate models together: a chat model, a classifier that decides what to do next, and an ASR model when the input is audio. IBM's Granite Switch 4.1 collapses all three into a single endpoint that occupies roughly the same VRAM as the base model alone. The catch is that you are now betting on one vendor's router and adapters, and the adapter-training tooling is still a promise.

Three adapters, one checkpoint, zero extra VRAM

Granite-Switch-4.1-8B and -30B ship as single checkpoints that compose the base model with a predefined set of adapters (IBM calls them "batteries included"). Per the Hugging Face announcement, each checkpoint includes:

  • A Decision aLoRA for general-purpose routing, e.g. the v1/systemone endpoint.
  • A Guardian aLoRA that, on average, outperforms Jev 1.13, Microsoft-Decision-1, GPT-Luna-Decisions, and Clef-Flash on safety-related decision tasks.
  • Granite-Speech-5.0-Turbo-CTC, which the post describes as the fastest ASR model on the OpenASR Leaderboard, embedded for transcription.

Because the adapters are built as cache-efficient Activated LoRAs, the KV cache from a base-model generation is reused for any subsequent adapter call. IBM says all three technologies take "approximately the same amount of VRAM on a GPU as the base model would by itself" — more capabilities without more GPUs. The 8B variant is listed at ~9B parameters and the 30B at 30B, so the 8B is the one that fits a single consumer card.

Where the speed claim actually comes from

The "switch between chat, decision, and speech requests at blazing fast speeds" line is not hand-waving. It rests on two concrete pieces: IBM's recent vLLM contributions to speed up low-rank-adapter inference, plus a co-located ASR encoder, with a shared prefix cache across request types. There is also a running HF Space where granite-switch-4.1-8b transcribes audio, generates summaries, screens for harmful content, and tests 15 other user-defined flags on a single endpoint in real time.

In a "race" the post describes, Granite-Switch-4.1-30B (frozen Granite base + 4 specialized aLoRAs) completed a real-world 5-step RAG workflow faster than Jev-27B (frozen Qwen base + 1 generic decision LoRA). And in a Doom-playing demo, a custom 3B Granite Switch model made decisions at 60–100 decisions/second versus Jev's reported 10, while also narrating gameplay — numbers IBM reports, so treat them as vendor claims.

The drop-in angle and the missing tooling

The most useful bit for an architect is the API story. The model runs in vLLM via a plug-in and has native inference in llama.cpp, and it can "seamlessly swap in for /v1/systemone API calls anywhere they're used, while also servicing traditional chat calls, all with a shared prefix cache." If you have already standardized on /v1/systemone (Typesafe's decision-model convention), that is a one-line endpoint change rather than a rewrite.

You are not locked into IBM's predefined mix. The post frames the real opportunity as community composition: anyone can build their own Granite Switch by mixing adapters from the Granite Libraries collections (granitelib-core-r1.0, granitelib-guardian-r1.0, etc.). But the tooling to make that easy is not out yet — an Adapter SDK for training adapters in Mellea, plus tutorials, is only "planned" for the coming weeks. Until that lands, composing a custom adapter set is manual and the ecosystem is thin.

When I'd reach for it

I would evaluate this when I have a voice-in, decisions-in-the-middle agent and I want to cut integration surface and the round trips between a chat service and a classifier service — the shared KV/prefix cache is the real win, not the parameter count. I'd start with the 8B on a single GPU and watch two things: the Adapter SDK for Mellea actually shipping (that decides whether custom adapters are realistic), and real latency numbers from the vLLM plug-in rather than the demo Space. The Guardian safety claims and the RAG/Doom speedups are vendor-reported; I'd want to reproduce them on my own task mix before I retire a dedicated classifier.

Sources: