⚡ LLM Inference - calleum · Scour

LLM Inference Engineering Room — Part 3: The Orchestration Layer

🧠LLM Training Blog

vimal-dwarampudi.medium.com·

KaiFelixBennett/gemma4-turboquant-rdna4: Run Gemma-4-31B at full 256K context on a $1,400 AMD RDNA4 GPU (gfx1201): TurboQuant KV cache + HIP-graph-safe Flash-Attention for llama.cpp, fully measured on real hardware.

🐧Linux Kernel Code

github.com··Hacker News

The Memory Problem is Solved: How Google’s Memory Caching Makes RNNs Smart Again

🌐Distributed Systems Blog

Running LLM Inference on Kubernetes: What It Actually Takes

🧠LLM Training Blog

fairwinds.com·

Anatomy of a high-performance EP kernel

⚙️Systems Programming Blog

fergusfinn.com··Hacker News

heterodoxin/graphkv: Graph-guided KV cache compression for memory-efficient LLM inference.

🧠LLM Training Code

github.com··r/LocalLLaMA

Less-relevant results

AMD's Lemonade SDK For Local AI Adds NVIDIA CUDA Support

Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

📄CS Papers Academic

From GPU to Token: The 8-Layer Observability Stack for AI Infrastructure

⚙️Systems Programming Blog

Making Local LLM Go Brrr

🧠LLM Training

seanpedersen.github.io·

huawei-csl/KVarN: KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

🐧Linux Kernel Code

github.com··Hacker News

147th airhacks tv: Local LLMs, LightMetal, ZSmith Agents, AI Rails, Saving Tokens

⚡Zig Blog

adambien.blog·

Youssof Altoukhi (@Youssofal_)

xcancel.com··r/LocalLLaMA

BeeLlama.cpp DFlash on Strix Halo: 2.7x Gemma 31B, But MTP Is Still Faster

sleepingrobots.com·

Report: GKE Inference Gateway delivers up to 92% faster AI responses

🕸️axum Blog

cloud.google.com··Hacker News

Making LLMs faster and more efficient across multiple languages

🧠LLM Training

techxplore.com·

defai-digital/ax-engine: Apple Silicon LLM runtime supporting Gemma 4 and Qwen 3.6 MTP modes

🧠LLM Training Code

github.com··Hacker News

China's Xiaomi MiMo Is Now 15X Faster Than ChatGPT and Claude (4 minute read)

🧠LLM Training News

MoQ GGUFs and GSQ: Low-Bit GGUFs Are About to Get Much Better

🧠LLM Training News Blog

kaitchup.substack.com··r/LocalLLaMA

bigattichouse/packed-twin-inference: PTI achieves ~2× throughput using a single quantized model (Q5_K_M or better) by running 4 generation streams in one batched decode call. The GPU loads model weights once per step and produces 4 predictions simultaneously. KV cache overhead is ~0.8 GiB total for all 4 streams. No draft model. No quality loss

⚡Zig Code

github.com··r/LocalLLaMA

Log in to enable infinite scrolling