Mistral Large 4: 1T Parameters at $1.36/$4.18 per Million Tokens
Mistral Large 4 public preview: 1T-parameter multimodal MoE, 49B active, $1.36/$4.18 per million tokens, weights promised by end of October.
#inference
10 posts
Beam Puts 80.1 on Terminal-Bench in 23B Active Parameters
Reflection's Beam is a 501B MoE with 23B active parameters, 80.1 on Terminal Bench 2.1 and 80.9 on SWE-bench Verified. Apache 2.0 weights this month.
A 125B Model Now Writes 94 tok/s on a 12GB RTX 5070
Strata runs Qwen3.8-Flash-Next, a 125B MoE, on 12GB consumer cards: 94 tok/s write speed at Q2_0 on an RTX 5070, with an OpenAI/Anthropic-compatible server on localhost.
Ember-1 Cuts 40% of Kimi K3's Tokens at Equal Quality
Fireworks Research's Ember-1 is a Kimi K3 derivative that drops 35-50% of reasoning tokens at matching accuracy, priced against K3's $3/$15 per M token rates.
TPU 8i Adds 288GB HBM and 384MB SRAM for Agent Inference
Google's eighth-gen TPUs split into TPU 8t for training (121 ExaFlops, 9,600 chips) and TPU 8i for inference (288GB HBM, 80% better perf-per-dollar).
NVIDIA's Custom HBM Signals Memory Bandwidth Is the New Constraint
NVIDIA's NVHBM moves memory control to the HBM stack itself, not the GPU. This architectural shift reveals memory bandwidth, not compute cores, is the real scaling bottleneck for AI inference.
vLLM 0.27 Release: New CondensePyramid V1 Attention Kernel and Multi-Modal Token Limits
vLLM 0.27 adds CondensePyramid V1 attention for multi-modal long context, new token-tier limits on templates, faster ASR CPU preprocessing via multi-threading, and CPU W4A16 INT4 MoE support.
1T-Param Model Hits 1000 tok/s Without Custom Silicon
TileRT 1.5 runs a 1T-param Xiaomi model at 1000+ tokens/s on a single 8x B200 node. Here's the exact stack, commands, and where it fits for low-latency serving.
vLLM 0.27 Removes First-Request Stalls — But Torch 2.13 Is Breaking
vLLM 0.27.0 (Aug 10) ships JIT warmup and runner-owned Triton warmup that eliminate first-request compile stalls, plus Kimi K3 support — but the torch 2.13.0 upgrade is a breaking change that needs migration testing before you bump.
Retune Draft Models Without Restarting vLLM
vLLM v0.26.0 adds runtime draft-weight updates, so you can retune speculative decoding without restarting the engine and flushing the KV cache.