🔧 Model Serving - meghanipankaj5scour

🧠LLM Internals Academic

arxiv.org··Hacker News

Inferoa AI harness claimed 90% cache savings. We ran it and measured 97.8%

🧠LLM Internals

zozo123.github.io··Hacker News

PagedAttention vs Traditional KV Cache: How vLLM Reinvented GPU Memory for LLM Inference

🖥️Systems Programming Blog

medium.com

Less-relevant results

DiffusionGemma: The Developer Guide

🧠LLM Internals Blog

developers.googleblog.com·

How we fight GPU scarcity without compromise

🏗️System Design Blog

equixly.com··Hacker News

Big Blue’s Redbook on Storage Scale KV Cache management

💾CPU Architecture News

blocksandfiles.com·

The Inference Alpha: Maximizing Frontier Models on AMD

🖥️Systems Programming Blog

digitalocean.com·

DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time - Huawei, GB300 NVL72, MI355X, B200

🖥️Systems Programming News

newsletter.semianalysis.com

··Hacker News

Speculators v0.5.0: DFlash support and online training

🔨Compilers

developers.redhat.com·

2x GH200 for LLM inference, Part 2: vLLM, DeepSeek V4 Flash, and MTP

🖥️Systems Programming Blog

dnhkng.github.io·

AMD's Lemonade SDK For Local AI Adds NVIDIA CUDA Support

🔨Compilers

phoronix.com·

google/gemma-4-31B-it · fix: chat template — null handling, reasoning preservation, turn-tag balance, input validation

🦀Rust

huggingface.co··r/LocalLLaMA

Magenta RealTime 2: Open and Local Live Music Models

🧠LLM Internals

magenta.withgoogle.com··Hacker News, Hacker News, r/LocalLLaMA

AI Serving Platform That Adapts to Your Model

🏗️System Design Blog

databricks.com·

libertywing/FlashMemory-Deepseek-V4: FlashMemory DS-V4 Retriever: a lightweight retriever that sparsifies DeepSeek-V4 CSA KV-cache. Weights available on Hugging Face.

💾CPU Architecture Code

github.com·

From GPU to Token: The 8-Layer Observability Stack for AI Infrastructure

🖥️Systems Programming Blog

jimmysong.io·

RKSC: Reasoning-Aware KV Cache Sharing and Confident Early Exit for Multi-Step LLM Inference

🧠LLM Internals Academic

arxiv.org·

OpenCV Introduces New DNN Inference Engine

⚙️C++

i-programmer.info·

Running LLM Inference on Kubernetes: What It Actually Takes

🧠LLM Internals Blog

fairwinds.com·

harshuljain13/llm-inference-at-scale: A Practitioner handbook for production llm serving.

Breaking the Ice: Analyzing Cold Start Latency in vLLM

Inferoa AI harness claimed 90% cache savings. We ran it and measured 97.8%

PagedAttention vs Traditional KV Cache: How vLLM Reinvented GPU Memory for LLM Inference

DiffusionGemma: The Developer Guide

How we fight GPU scarcity without compromise

Big Blue’s Redbook on Storage Scale KV Cache management

The Inference Alpha: Maximizing Frontier Models on AMD

DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time - Huawei, GB300 NVL72, MI355X, B200

Speculators v0.5.0: DFlash support and online training

2x GH200 for LLM inference, Part 2: vLLM, DeepSeek V4 Flash, and MTP

AMD's Lemonade SDK For Local AI Adds NVIDIA CUDA Support

google/gemma-4-31B-it · fix: chat template — null handling, reasoning preservation, turn-tag balance, input validation

Magenta RealTime 2: Open and Local Live Music Models

AI Serving Platform That Adapts to Your Model

libertywing/FlashMemory-Deepseek-V4: FlashMemory DS-V4 Retriever: a lightweight retriever that sparsifies DeepSeek-V4 CSA KV-cache. Weights available on Hugging Face.

From GPU to Token: The 8-Layer Observability Stack for AI Infrastructure

RKSC: Reasoning-Aware KV Cache Sharing and Confident Early Exit for Multi-Step LLM Inference

OpenCV Introduces New DNN Inference Engine

Running LLM Inference on Kubernetes: What It Actually Takes