💾 Prompt Caching - emschwartz

google/gemma-4-31B-it · fix: chat template — null handling, reasoning preservation, turn-tag balance, input validation

⚡Fast AI Inference

huggingface.co··r/LocalLLaMA

Prompt Caching

🕳LLM Vulnerabilities

pub.towardsai.net

How we fight GPU scarcity without compromise

🏗️LLM Infrastructure Blog

equixly.com··Hacker News

huawei-csl/KVarN: KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

🏗️LLM Infrastructure Code

github.com··Hacker News

Breaking the Ice: Analyzing Cold Start Latency in vLLM

🏗️LLM Infrastructure Academic

arxiv.org··Hacker News

Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

🧠LLM Inference Academic

arxiv.org·

Nvidia DGX Spark GB10 – AI Models and Guide with vLLM and Autonomous Script

🧠Inference Serving Code

github.com··Hacker News

Prompt Caching on Claude: Cut Input Costs 78% (The Math Nobody Writes Down)

💻Claude Code

pub.towardsai.net

STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control

🧠LLM Inference Academic

arxiv.org·

KJLdefeated/RL.cu: RLVR training for LLM in CUDA/C++

🏗️LLM Infrastructure Code

github.com··Hacker News

From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs

🤖AI Academic

arxiv.org·

harshuljain13/llm-inference-at-scale: A Practitioner handbook for production llm serving.

🧠LLM Inference Code

github.com··Hacker News

mirkolenz/llmhop: Tiny, stateless Go router that dispatches OpenAI-compatible requests to single-model vLLM and sglang backends with zero external dependencies

🤖AI Code

github.com··Hacker News

RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

🏗️LLM Infrastructure Academic

arxiv.org·

Does anyone know what PCIe mode was used for these benchmarks?

🏗️LLM Infrastructure Code

github.com··r/LocalLLaMA

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

🏗️LLM Infrastructure Academic

arxiv.org··Hacker News

No more posts from emschwartz's subscribed feeds.

Scour all 25257 feeds Learn more about Feeds

heterodoxin/graphkv: Graph-guided KV cache compression for memory-efficient LLM inference.

RKSC: Reasoning-Aware KV Cache Sharing and Confident Early Exit for Multi-Step LLM Inference

DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time - Huawei, GB300 NVL72, MI355X, B200

Inferoa AI harness claimed 90% cache savings. We ran it and measured 97.8%

google/gemma-4-31B-it · fix: chat template — null handling, reasoning preservation, turn-tag balance, input validation

Prompt Caching

How we fight GPU scarcity without compromise

huawei-csl/KVarN: KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

Breaking the Ice: Analyzing Cold Start Latency in vLLM

Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

Nvidia DGX Spark GB10 – AI Models and Guide with vLLM and Autonomous Script

Prompt Caching on Claude: Cut Input Costs 78% (The Math Nobody Writes Down)

STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control

KJLdefeated/RL.cu: RLVR training for LLM in CUDA/C++

From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs

harshuljain13/llm-inference-at-scale: A Practitioner handbook for production llm serving.

mirkolenz/llmhop: Tiny, stateless Go router that dispatches OpenAI-compatible requests to single-model vLLM and sglang backends with zero external dependencies

RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

Does anyone know what PCIe mode was used for these benchmarks?

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving