DeepSeek has published DeepInfer on GitHub under Apache-2.0. The README calls it "the high-performance inference framework that powers DeepSeek's large-scale production services." Before you plan a deployment, read the scope: this is an offline inference release, it supports one model, and it targets two hardware families. It is a reference for how DeepSeek serves its model, not a drop-in replacement for your current serving stack.
What the release covers
The only supported model is DeepSeek-V4.1-Flash. The default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017, which I covered in the Terminal-Bench 4.0 post. It downloads into the Hugging Face cache on first run.
Supported hardware:
- NVIDIA SM100 GPUs
- Huawei Ascend 950 NPUs
The feature list is long. Optimized kernels, model and sampler runtime. Attention DP / balanced CP and MoE EP + EPLB for parallelism. DSpark with confidence-scheduled verification. SWA bounded replay. A zero-overhead asynchronous batch scheduler. Batch-invariant inference, deterministic sampling and structured outputs.
The kernels are named: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, plus custom TileLang kernels. The README also says DeepInfer was originally built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM.
Environment variables that control the features
The configuration is almost entirely environment variables. This is the part worth bookmarking.
| Feature | Setting | Notes |
|---|---|---|
| DSpark / confidence-scheduled verification | DEEPINFER_ENABLE_MTP_VARLEN=1 |
Draft tokens, with confidence estimates scheduling verification |
| Max draft length | DEEPINFER_USE_NEXTN_DECODE |
Default: 5 |
| Decoder SWA bounded replay | DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact |
Replays only a bounded suffix |
| Replay suffix length | DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 |
Default V4.1 setting |
| Batch-invariant inference | DEEPINFER_BATCH_INVARIANT=1 |
NVIDIA only |
| Deterministic sampling | DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 plus a fixed seq_seed in SamplingParams |
NVIDIA only |
| Structured outputs | so_args={"schema_type": ..., "schema": ...} in SamplingParams |
Regex, JSON schema or EBNF grammar |
| Asynchronous scheduling | none | On by default |
Two details matter here. The retained suffix for SWA replay must cover the SWA window plus the maximum backtracking length, so don't lower the 140 without checking that constraint. And deterministic sampling means turning the speculative path off (DEEPINFER_ENABLE_MTP_VARLEN=0). You are choosing reproducibility over draft-token speedup. The same trade-off exists in other runtimes, as I wrote in the speculative decoding explainer.
Batch-invariant inference is the one I would reach for in evals and regression tests. It keeps results identical across different batch compositions, which is exactly what flaky benchmark comparisons need.
Build and run
Prerequisites: Linux, GCC >= 13.0, Python 3.12, and CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). Start from an empty Python 3.12 environment:
# Use `ascend` instead for the Ascend platform
# Keep it exported in the environment where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.py
To run from a local copy of the weights, set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017.
What is missing
The README is explicit about what a complete service still needs, and none of it ships here:
- A KV cache system
- VE/Prefill/Decode disaggregation (the README says most of the required engine functionality is already implemented)
- Load balancing across data-parallel replicas for both prefill and decode
That is the layer that turns an engine into a service. If you wanted to run this as an endpoint, you would be building it yourself.
The repo also includes prefill and decode benchmark charts. The README says they were measured on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and points to run_sweep.sh to reproduce them. I am not quoting throughput figures here because the README text I read does not state them as numbers. Run the sweep yourself. Note that the decode setup is 64 GPUs, so those numbers describe a cluster deployment, not a single box.
Who should care
If you run a few consumer GPUs, this is not for you: the hardware list is SM100 and Ascend 950 only. If you operate SM100 or Ascend clusters and self-host DeepSeek, it is the first look at the vendor's own serving stack. Even if you never deploy it, the config surface (bounded SWA replay, confidence-scheduled verification, batch invariance) shows what DeepSeek tunes in production. It is a useful checklist to compare against vLLM or SGLang.
Sources: