Google’s TurboQuant Compression May Support Faster Inference, Same Accuracy on Less Capable Hardware (opens in new tab)

Google Research unveiled TurboQuant, a novel quantization algorithm that compresses large language models’ Key-Value caches by up to 6x. With 3.5-bit compression, near-zero accuracy loss, and no retraining needed, it allows developers to run massive context windows on significantly more modest hardware than previously required. Early community benchmarks confirm significant efficiency gains. By Bruno Couriol