Rapid-MLX, a community-driven MLX inference engine, just shipped a new round of optimizations that fundamentally changes how large MoE models behave on Apple Silicon. The headline number: DeepSeek V4 Flash 158B-A13B at 2-bit DQ runs at 56 tokens per second on a single Mac, with day-zero support baked in.
That is not a theoretical ceiling. It is a measured, sustained decode rate that crosses the threshold from demo curiosity to usable for long-horizon agentic loops.
The Benchmark Deltas
Rapid-MLX benchmarks against the standard mlx-lm serve and Ollama stacks on a suite of recent open-weight models. The gains concentrate in KV cache management and hybrid architecture handling.
| Model | Rapid-MLX | Best Alternative | Speedup |
|---|---|---|---|
| Phi-4 Mini 14B | 180 tok/s | 77 (mlx-lm) / 56 (Ollama) | 2.3× / 3.2× |
| GPT-OSS 20B | 127 tok/s | 79 (mlx-lm serve) | 1.6× |
| Qwen3.5-9B | 108 tok/s | 41 (Ollama) | 2.6× |
| Kimi-Linear-48B | 94 tok/s | — (only engine) | — |
| DeepSeek V4 Flash 158B-A13B (2-bit DQ) | 56 tok/s | — (only MLX engine, day-0) | — |
The 158B-A13B model activates 13B parameters per token while holding 158B in memory. Standard MLX runners struggle to schedule the expert routing efficiently, causing decode to stall or swap to unified memory. Rapid-MLX's scheduler keeps the active expert set pinned and routes the 13B computation stream without dropping the larger sparse weights into swap.
How the KV Cache Snapshots Work
The biggest throughput leap on mid-sized models comes from a structural change in how the KV cache is handled. Rapid-MLX ships with pure KV cache snapshots and DeltaNet state snapshots for hybrid RNN + attention models.
The effect is immediate on Time-To-First-Token (TTFT):
| Model | Rapid-MLX (cached) | mlx-lm serve | Delta |
|---|---|---|---|
| Hermes-3-Llama 8B | 0.10s | 0.18s | 1.8× |
| Devstral-Small-2 24B | 0.13s | 0.38s | 2.9× |
| Mistral Small 24B | 0.13s | 0.38s | 2.9× |
| GLM-4.5-Air | 0.14s | 0.47s | 3.4× |
By snapshotting the pure KV cache and hybrid RNN states, the engine avoids recompute on prompt rotation. You get the speedup on subsequent calls or when chaining tool calls in an agent loop, where context window management is the bottleneck, not raw decode.
The 2-Bit DQ Quantization
DeepSeek V4 Flash at 158B total parameters normally requires ~314 GB of memory at BF16. Rapid-MLX ships with a 2-bit DQ (Dynamic Quantization) variant that compresses the weights to ~31 GB, plus KV cache overhead. This fits comfortably on a 64 GB Mac Studio or Mac Pro. The 2-bit DQ format drops the token acceptance rate slightly compared to 8-bit, but the throughput delta is massive. At 56 tok/s, you are generating a full coding patch in roughly 30–40 seconds.
Where It Breaks
Rapid-MLX is a third-party engine. You are not running the official mlx-lm or mlx-community reference builds. The tradeoff is clear:
- You lose official support and standardized API shapes. The engine requires specific launch flags and model paths.
- Hybrid models (RNN + attention) like Kimi-Linear and DeepSeek V4 Flash need explicit state snapshot handling. If your prompt structure changes the RNN context, you may see state drift.
- Quantization artifacts at 2-bit DQ show up on highly constrained tasks. If you need deterministic output for API schema enforcement, test explicitly before shipping.
How to Run It
Clone the Rapid-MLX repository, activate the environment, and pull the 2-bit DQ variant for DeepSeek V4 Flash. The invocation is straightforward but requires the correct model ID:
git clone https://github.com/raullenchai/Rapid-MLX.git
cd Rapid-MLX
source bin/activate
python -m mlx_vlm.chat --model rapid-mlx/deepseek-v4-flash-158b-2bit-dq
For benchmarking your own setup, run the built-in inference bench or pull the Qwen3.6-35B-A3B variant to validate the 95 tok/s number against your own hardware.
The Takeaway
The 150B parameter wall was supposed to require server-grade clusters. Rapid-MLX's engine optimizations prove that with the right scheduler, hybrid state snapshots, and aggressive quantization, consumer Macs can run frontier-tier MoE models at usable speeds. The 56 tok/s number is the benchmark that matters. If your agent loop needs a 150B reasoning model and you don't have an H100, this is the path.
Sources: