Recency/Frequency Adaptive KV Caching for Large Language Model Serving

Key-value (KV) caching is a powerful technique for accelerating large language model inference and generation. Inference workloads are large and diverse, which makes them difficult to cache effectively. Existing cache management strategies adopt the least-recently-used policy for evicting cache bloc…

View Post on Scour ↗

More from arXiv on Scour.