🧠 LLM Inference - emschwartz · Scour

DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference 🏗️LLM Infrastructure

The Inference Economy: Token Use 💰Tokenomics

frontierai.substack.com·11h·Substack

Adaptive Thinking: Large Language Models Know When to Think in Latent Space 🏗️LLM Infrastructure

machinelearning.apple.com·2d

AmSach/kvquant: Drop-in KV cache compressor for local LLM inference - Run 70B models on 8GB RAM 🏗️LLM Infrastructure

github.com·17h·DEV

DeepSeek-V4 on Day 0: From Fast Inference to Verified RL with SGLang and Miles 🧠Inference Serving

lmsys.org·5d·Hacker News

Speculative Decoding vs MoE: 3.2x Cost Gap on Llama 3 📊Model Serving Economics

tildalice.io·3d

Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity 🏗️LLM Infrastructure

arxiv.org·2d·Hacker News

shreyansh26/Speculative-Decoding: Speculative Decoding Implementations: EAGLE-3, Medusa-1, PARD, Draft Models, N-gram and Suffix Decoding from scratch 📊Model Serving Economics

github.com·4d·r/LLM, r/LocalLLaMA

PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference 🏗️LLM Infrastructure

DAK: Direct-Access-Enabled GPU Memory Offloading with Optimal Efficiency for LLM Inference 🏗️LLM Infrastructure

Efficient, VRAM-Constrained xLM Inference on Clients 🏗️LLM Infrastructure

QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention 🔬RaBitQ

DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference 🏗️LLM Infrastructure

Network Edge Inference for Large Language Models: Principles, Techniques, and Opportunities 📱Edge AI Optimization

Scalable Inference Architectures for Compound AI Systems: A Production Deployment Study 🏗️LLM Infrastructure

Select to Think: Unlocking SLM Potential with Local Sufficiency 🏗️LLM Infrastructure

Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inference 🏗️LLM Infrastructure

Exploring the Efficiency of 3D-Stacked AI Chip Architecture for LLM Inference with Voxel 🏗️LLM Infrastructure

Anchored Variational Inference for Personalized Sequential Latent-State Models 🏗️LLM Infrastructure

One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinement 🏗️LLM Infrastructure

Log in to enable infinite scrolling