🧠 LLM Inference - emschwartz · Scour

SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference 🏗️LLM Infrastructure

The Inference Economy: Token Use 💰Tokenomics

frontierai.substack.com·5h·Substack

Adaptive Thinking: Large Language Models Know When to Think in Latent Space 🏗️LLM Infrastructure

machinelearning.apple.com·1d

DeepSeek-V4 on Day 0: From Fast Inference to Verified RL with SGLang and Miles 🧠Inference Serving

lmsys.org·4d·Hacker News

DeepSeek V4 in vLLM: Efficient Long-Context Attention 🏗️LLM Infrastructure

vllm-website-pdzeaspbm-inferact-inc.vercel.app·6d·Hacker News

shreyansh26/Speculative-Decoding: Speculative Decoding Implementations: EAGLE-3, Medusa-1, PARD, Draft Models, N-gram and Suffix Decoding from scratch 📊Model Serving Economics

github.com·4d·r/LLM, r/LocalLLaMA

DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference 🏗️LLM Infrastructure

Inside the Softmax Bottleneck: Engineering Hardware-Aware Attention Mechanisms 🏗️LLM Infrastructure

pub.towardsai.net

·6d

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities 📱Edge AI Optimization

DAK: Direct-Access-Enabled GPU Memory Offloading with Optimal Efficiency for LLM Inference 🏗️LLM Infrastructure

Efficient, VRAM-Constrained xLM Inference on Clients 🏗️LLM Infrastructure

DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference 🏗️LLM Infrastructure

Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity 🏗️LLM Infrastructure

arxiv.org·1d·Hacker News

Select to Think: Unlocking SLM Potential with Local Sufficiency 🏗️LLM Infrastructure

Exploring the Efficiency of 3D-Stacked AI Chip Architecture for LLM Inference with Voxel 🏗️LLM Infrastructure

PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference 🏗️LLM Infrastructure

Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inference 🏗️LLM Infrastructure

QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention 🔬RaBitQ

Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns 🧩MoE

MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference 🏗️LLM Infrastructure

Log in to enable infinite scrolling