Tether is shipping TurboQuant KV-cache quantization with Vulkan support into its QVAC SDK (opens in new tab)

The latest release of qvac-fabric-llm.cpp, the inference engine of the QVAC Fabric LLM, features TurboQuant integration for resource management in long-running inference sessions. Tether adopts the technology as a path to better efficiency when running large language models on devices with limited compute resources. TurboQuant is Google’s response to the Key-Value (KV) Cache’s capacity expansion during routine inference, which can reach up to 8GB for a 262,000-token context session using a 4B...