⚡ Inference Optimization - moyutianzun

💾KV Cache Academic

arxiv.org·

KaiFelixBennett/gemma4-turboquant-rdna4: Run Gemma-4-31B at full 256K context on a $1,400 AMD RDNA4 GPU (gfx1201): TurboQuant KV cache + HIP-graph-safe Flash-Attention for llama.cpp, fully measured on real hardware.

💾KV Cache Code

github.com··Hacker News

Inferoa AI harness claimed 90% cache savings. We ran it and measured 97.8%

💾KV Cache

zozo123.github.io··Hacker News

Qwen 3.6 27B AutoRound GGUF, need your feedback

🔥PyTorch Internals

huggingface.co··r/LocalLLaMA

Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency

🔄Transformers News Blog

blog.google··Hacker News

Big Blue’s Redbook on Storage Scale KV Cache management

💾KV Cache News

blocksandfiles.com·

Pruned YOLOv8 ONNX INT8 Fails: 3 Fixes That Work

🔥PyTorch Internals Blog Discussion

tildalice.io·

MTP Isn't Always a Win: 1.95x on My 3090, but Speculative Decoding Is Hardware-Dependent

⚡CUDA Blog

bric.pe.kr··DEV

Less-relevant results

The Inference Alpha: Maximizing Frontier Models on AMD

💾KV Cache Blog

digitalocean.com·

HNSW vs LSH: How Elasticsearch hits 0.99 recall@10 at 15,000 QPS — and what it costs

🔍RAG Blog

elastic.co·

MoQ GGUFs and GSQ: Low-Bit GGUFs Are About to Get Much Better

💾KV Cache News Blog

kaitchup.substack.com··r/LocalLLaMA

DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time - Huawei, GB300 NVL72, MI355X, B200

💾KV Cache News

newsletter.semianalysis.com

··Hacker News

The latest Gemma 4 models use a training trick to slash their on-device memory footprint

🔄Transformers

androidauthority.com·

Intelligent inference scheduling with llm-d on Red Hat AI

💾KV Cache

developers.redhat.com·

Unsloth Gemma 4 QAT

⚡CUDA

unsloth.ai·

China's Xiaomi MiMo Is Now 15X Faster Than ChatGPT and Claude (4 minute read)

🔄Transformers News

decrypt.co··Hacker News

Here's a llama.cpp CLI Command builder.

🔧MLIR

llamabuilding.com··r/LocalLLaMA

The economics of speculative decoding

🎭Mixture of Experts Blog

fergusfinn.com··Hacker News

Running Qwen 35B MoE at 450k Context on a Single 32GB GPU

🎭Mixture of Experts

local-llm.utop.workers.dev··Hacker News

GGUF vs GPTQ vs AWQ: The Plain-English Guide to LLM Quantization (and Which One to Pick)

Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

KaiFelixBennett/gemma4-turboquant-rdna4: Run Gemma-4-31B at full 256K context on a $1,400 AMD RDNA4 GPU (gfx1201): TurboQuant KV cache + HIP-graph-safe Flash-Attention for llama.cpp, fully measured on real hardware.

Inferoa AI harness claimed 90% cache savings. We ran it and measured 97.8%

Qwen 3.6 27B AutoRound GGUF, need your feedback

Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency

Big Blue’s Redbook on Storage Scale KV Cache management

Pruned YOLOv8 ONNX INT8 Fails: 3 Fixes That Work

MTP Isn't Always a Win: 1.95x on My 3090, but Speculative Decoding Is Hardware-Dependent

The Inference Alpha: Maximizing Frontier Models on AMD

HNSW vs LSH: How Elasticsearch hits 0.99 recall@10 at 15,000 QPS — and what it costs

MoQ GGUFs and GSQ: Low-Bit GGUFs Are About to Get Much Better

DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time - Huawei, GB300 NVL72, MI355X, B200

The latest Gemma 4 models use a training trick to slash their on-device memory footprint

Intelligent inference scheduling with llm-d on Red Hat AI

Unsloth Gemma 4 QAT

China's Xiaomi MiMo Is Now 15X Faster Than ChatGPT and Claude (4 minute read)

Here's a llama.cpp CLI Command builder.

The economics of speculative decoding

Running Qwen 35B MoE at 450k Context on a Single 32GB GPU