22 posts
University of Waterloo's PAW system compiles fuzzy-function specs into 23MB LoRA adapters that match 32B model accuracy on a 600M interpreter—no API calls, no cloud dependency, 30 tok/s on MacBook M3.
NVIDIA cut RTX 50 supply by 20% and sidelined the 5070 Ti. Street prices are up 19%. Here's how it changes your local AI build.
Z.AI's open-weights GLM-5.2 hits 81% on Terminal-Bench, undercutting GPT-5.5 for a fraction of the cost. Here's the benchmark breakdown.
Google's open 26B diffusion model hits 700+ tok/s on consumer GPUs with day-zero vLLM support. Here's what the bidirectional architecture changes for local inference.
Seattle's data center ban and 3-year grid queues mean local inference and modular hardware are no longer optional.
The SEC filing exposes a nine-to-one burn-to-revenue ratio. Here is why local inference is shifting from a privacy play to a hard P&L requirement.
OpenAI's IPO filing ends the era of subsidized frontier AI. Here's what the pricing shift means for local inference, enterprise contracts, and your agentic stack.
NVIDIA's first consumer SoC brings 128 GB of unified memory and native CUDA to Windows laptops — here's what it means for running LLMs locally, and the catches.
Rapid-MLX's new engine optimizations push DeepSeek V4 Flash 158B-A13B to 56 tok/s on a Mac. Here are the exact numbers and the tradeoffs.
Liquid AI's LFM2.5-8B-A1B hits under 6GB RAM with 1.5B active parameters, native tool calling, and verified throughput across edge hardware.
A community llama.cpp fork squeezes 110 tok/s out of Qwen3.6-35B-A3B MTP on a 12GB card. Here are the exact flags and the VRAM trick to make it fit.
Multi-token prediction is merged into llama.cpp, delivering 1.4–2.2x throughput on Qwen3.6 with zero accuracy loss.
llama.cpp merged Multi-Token Prediction for Qwen3. Community benchmarks show 38→47 tok/s on RTX 3090 and 63→84 tok/s on RTX 5090 — no new hardware needed.
One llama.cpp flag—--spec-draft-p-min 0.75—turns MTP from a dud into a decode speed that holds flat across output lengths where DFlash falls apart.
The --spec-draft-p-min filter in llama.cpp PR #22397 rescues MTP for Qwen3.6-27B: 48.9 tok/s vs 29 tok/s at 2000 tokens on a 24GB card.
vLLM 0.21.0 shipped Friday with a quiet fix: thinking_token_budget was being silently ignored when MTP speculative decoding was enabled. If you serve reasoning models with spec decode, you have been paying for it.
llama.cpp renamed the MTP flag on May 13. The old --spec-type mtp is silently ignored. If your tok/s dropped from 140 to 70 you are likely running without speculative decoding.
The same Qwen3.6-27B that ran at 70 tokens/sec on a 4090 in January was running at 140 tokens/sec by April. Nothing changed about the model. Speculative decoding moved from research curiosity to default. Here is what it actually does.
Three model releases in three weeks moved local AI from 'good enough for hobbies' to 'good enough for production'. Here's what changed and why it matters.
A practical guide to running Qwen3.6-27B on consumer hardware in 2026 — memory requirements per quant level, recommended runners, and the MTP trick that doubles your tokens per second.
Qwen3.6-27B running locally now scores within 10 points of frontier closed models on SWE-bench Verified. The benchmark table, lined up side by side.
Google released Gemma 4 on April 2, 2026 — four variants from 2B to 31B, with 256K context, native vision and audio, and Apache 2.0 licensing. Here's what it's for, where it fits, and how to run it.