🧠 LLM Inference - emschwartz · Scour

LLM Inference 🏗️LLM Infrastructure

iop.systems·49m

SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips 🏗️LLM Infrastructure

supercomputing-system-ai-lab.github.io·2d·Hacker News

chiennv2000/orthrus: Fast, lossless LLM inference via dual-view diffusion decoding. 🏗️LLM Infrastructure

github.com·5d·Hacker News

SpecSA: Bridging Speculative Decoding and Sparse Attention for Efficient LLM Inference 🏗️LLM Infrastructure

InferenceBench: A Benchmark for Open-Ended Inference Optimization by AI Agents 🏗️LLM Infrastructure

inferencebench.ai·4h·Hacker News

Your LLM Server Is Wasting 80% of Its GPU Memory — Here’s How vLLM Fixes That 🏗️LLM Infrastructure

pub.towardsai.net

·1d

I tried 4 LLM speedup techniques on CPU. Three made it slower. ⚙️Mechanical Sympathy

deemwar-products.github.io·8h·Hacker News

https://www.together.ai/blog/coding-agent-benchmarks 💻Coding Agents

together.ai·5d

KV Cache Is Becoming the Memory Hierarchy of Inference 💾Prompt Caching

touchdown-labs.com·2d

GPU Memory Math for LLMs: Formula That Tells You What Fits on Your GPU 🏗️LLM Infrastructure

theahmadosman.substack.com·6h·Substack, r/LocalLLaMA

Benchmarking llama.cpp's brand-new MTP support on Strix Halo ⚡Glommio

calebcoffie.com·2d·Hacker News

How LLM Inference Works 🏗️LLM Infrastructure

arpitbhayani.me·6d·Hacker News

DrBearJew/llama.cpp at tbq4-rdna3-experiment 🏗️LLM Infrastructure

github.com·6d·r/LocalLLaMA

FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration 📊Model Serving Economics

Let AI Agents Write Your Serving Stack with VibeServe 🏗️LLM Infrastructure

syfi.cs.washington.edu·6d·Hacker News

VeriCache: Turning Lossy KV Cache into Lossless LLM Inference 🏗️LLM Infrastructure

RedToasty/llama.cpp_qts: Fixing --split-mode tensor, with different KV cache quantization types. 🏗️LLM Infrastructure

github.com·3d·r/LocalLLaMA

Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding 🔢BitNet Inference

I've updated my glorified Llama fork (LLM Inference Server) for P40's to utilise MTP + TurboQuant + DFlash 🏗️LLM Infrastructure

github.com·4d·r/LocalLLaMA

OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization 🔬RaBitQ

Log in to enable infinite scrolling