A 1-trillion-parameter model generating over 1000 tokens per second on a single 8-GPU node — no custom silicon, no TPU, no bespoke interconnect. The result comes from TileRT, a tile-level inference runtime from Tile AI, working in collaboration with Xiaomi MiMo (TileRT, GitHub; MiMo, June 2026).

The catch is the hardware: the official 1000 TPS benchmark is for MiMo-V2.5-Pro-UltraSpeed on one 8× NVIDIA B200 node. If your inference fleet is not B200, this is not directly portable. But the architecture is instructive, and the v0.1.5 release adds real production tooling worth knowing about.

The tile-level approach

TileRT does not compete with vLLM on throughput. It optimizes for responsiveness — millisecond-level time per output token (TPOT) — which matters for high-frequency trading, interactive agents, real-time coding assistants, and long-running agentic loops where a 200 ms token stall breaks the user experience (TileRT README).

The mechanism: a compiler decomposes LLM operators into fine-grained tile-level tasks. The runtime then dynamically reschedules compute, I/O, and communication across devices in an overlapped manner, minimizing GPU idle time. The underlying compiler techniques are being gradually integrated into TileLang and TileScale, both open-source.

That is the same class of trick that made speculative decoding useful in 2026, but applied at the kernel scheduling level rather than the token prediction level. You get the latency reduction without needing a separate draft model.

v0.1.5: PD disaggregation behind an OpenAI endpoint

The July 14 v0.1.5 release adds prefill-decode (PD) disaggregation: vLLM handles prefill, TileRT handles decode, and the whole thing sits behind a standard OpenAI-compatible HTTP endpoint (TileRT v0.1.5 release notes). Supported on GLM-5/5.1 and DeepSeek-V3.2.

The connector, decode server, and router all ship inside the tilert PyPI wheel under tilert.pd_vllm. No vLLM fork or patch is needed; the connector loads through vLLM's standard kv_connector_module_path V1 interface. You can route latency-critical requests to the TileRT decode pool and leave bulk traffic on native vLLM decode.

The pinned environment is strict. From the README:

Component Pinned version
GPU 8× NVIDIA B200
CUDA 13.2 runtime
Python 3.12
PyTorch 2.11.0+cu130
transformers 4.46.3

The recommended path is the pre-built Docker image:

docker pull ghcr.io/tile-ai/tilert:cu132-latest

pip install tilert==0.1.5.post1

Other Python, CUDA, or PyTorch combinations are explicitly untested. If your stack does not match, you are on your own.

Weight conversion and the MTP flag

Starting with v0.1.3, you no longer download pre-converted weights. You pull the official Hugging Face checkpoint and run the bundled converter, which shards the weights into 8 per-device files with keys suffixed *_dev_{0..7}:

python -m tilert.models.preprocess.weight_converter \
  --model_type deepseek-v32 \
  --model_dir "/path/to/DeepSeek-V3.2" \
  --save_dir "/path/to/DeepSeek-V3.2-TileRT"

GLM-5/5.1 uses --model_type glm-5.

Multi-Token Prediction is a first-class flag. Enable it with --with-mtp on the CLI or with_mtp=True in the Python API. The README reports a mean accepted length of 2.77 under MTP (min 1, max 4), meaning the model accepts roughly 2.8 tokens per forward pass on average.

One operational constraint: v0.1.5 ships two independent backend libraries (libtilert_dsv32.so and libtilert_glm5.so) and loads exactly one per Python process. You cannot run DeepSeek-V3.2 and GLM-5 in the same interpreter. Run them in separate processes or containers.

Where this lands in your stack

TileRT is not a general-purpose inference server. It is a decode-optimized engine for a specific hardware tier (B200) and a specific latency profile (single-request, low-batch, millisecond TPOT). I would reach for it when I have a latency-sensitive agent loop — an autonomous coding agent, a real-time trading signal generator, an interactive assistant with sub-100 ms token expectations — and I have B200s sitting idle or underutilized in a batch-serving configuration.

The 1000 TPS number on a 1T model is the headline, but the practical takeaway for most practitioners is the PD disaggregation pattern: vLLM prefill + TileRT decode behind a standard OpenAI endpoint, with no fork. That pattern is what you would copy into your own serving architecture even if you never install TileRT. The tile-scheduling trick is the reason it works, and the compiler pieces are landing in open-source TileLang and TileScale, so the technique will diffuse into other runtimes.

The B200 pinning is the real barrier. If you are on H100 or A100, the v0.1.5 wheel will not install. The project is actively evolving, and the README explicitly states the compiler techniques will be shared more broadly, but as of this writing the supported matrix is 8× B200 or nothing.

Production track record

This is not a research prototype. GLM-5.1-highspeed has been live on Z.ai (the inference platform behind Zhipu AI) since May 22, 2026, powered by TileRT, going from experimental to real production traffic (TileRT, May 2026). The Xiaomi MiMo collaboration pushed MiMo-V2.5-Pro-UltraSpeed past 1000 TPS on 1T parameters in June (MiMo blog). Two independent deployments, two different model families, both on the same runtime.

I would watch the TileLang and TileScale repositories next. The tile-scheduling compiler is the durable contribution here. The B200-specific wheel is the delivery vehicle. If the scheduling techniques land in a more general runtime, the latency gains will apply to hardware you already have.

Sources: