Two Leaps to 1000 Tokens/s on a 1T-Parameter Model: On Inference Systems, Execution Boundaries, and Co-Design (opens in new tab)

TileRT is a tile-level runtime engine that pushes the latency limits of large language models without compromising model size or quality.