🚀 Speculative Decoding - nayyara.airlangga · Scour

Multi-Drafter Speculative Decoding with Alignment Feedback ⚙️ML Compilers

arxiv.org·2d

LLM inference engine from scratch in C++ 🧠Inference Engineering

anirudhsathiya.com·4d·Hacker News

RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference 🧠Inference Engineering

vldb.org·1d

Speculative decoding made my local LLM actually usable 🧠Inference Engineering

xda-developers.com·4d

Inside LLM Inference: KV Cache, Prefill, and the Decode Bottleneck 🧠Inference Engineering

pub.towardsai.net

·1d

The Engine Behind Modern LLM Inference, Part 1: Continuous Batching, PagedAttention, and the End of… 🧠Inference Engineering

medium.com·1d

GPU Memory for LLM Inference: Why Llama-70B Doesn't Fit 🧠Inference Engineering

darshanfofadiya.com·4d·Hacker News

Apfel -- A CLI and http server for the on-device Apple Intelligence LLM 🧠Inference Engineering

discuss.privacyguides.net

·2d

KV Cache in LLM Inference: From PagedAttention (2023) to Reasoning Model Bottlenecks (2026) 💾KV Cache

medium.com·3d

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification 🧠Inference Engineering

arxiv.org·14h

Cryptographic Provenance for LLM Inference ⚙️MLOps

commitllm.com·6d·Hacker News

TurboQuant Explained: Extreme AI Compression for Faster, Cheaper LLM Inference and Vector Search 🗜️Quantization

medium.com

·5d

TurboQuant Is Quietly Solving LLM Inference’s Worst Memory Problem 🗜️Quantization

medium.com·5d

janit/viiwork: LLM inference load balancer optimized for AMD Radeon VII GPUs 🔴ROCm

github.com·5d·Hacker News

Inside the LLM Black Box: The True Architecture of Latency and Cost 🧠Inference Engineering

akanuri.medium.com·6d

Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC 🧠Inference Engineering

arxiv.org·14h

What is an AI Native Cloud? 📝Infrastructure as Code

together.ai·3d

AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention 🧠Inference Engineering

arxiv.org·14h

DualDiffusion: A Speculative Decoding Strategy for Masked Diffusion Models 🧠Inference Engineering

arxiv.org·2d

Valve: Production Online-Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate 🧠Inference Engineering

arxiv.org·14h

Loading more...