1T-Param Model Hits 1000 tok/s Without Custom Silicon
TileRT 1.5 runs a 1T-param Xiaomi model at 1000+ tokens/s on a single 8x B200 node. Here's the exact stack, commands, and where it fits for low-latency serving.
#tilert
1 post
1 post
TileRT 1.5 runs a 1T-param Xiaomi model at 1000+ tokens/s on a single 8x B200 node. Here's the exact stack, commands, and where it fits for low-latency serving.