Cache-Resident LLM Inference in GB-Scale Last-Level Caches

Large language model (LLM) inference is increasingly dominated by data movement across the memory hierarchy. Recent 3D-stacked cache technologies have enabled GB-scale last-level caches in modern server CPUs, making it possible to keep reusable model weights on chip and exploit cache bandwidth and l…

View Post on Scour ↗

More from arXiv on Scour.