Continuous Batching

⚡ Continuous BatchingSpecific

dynamic batching, vLLM, inference throughput, iteration-level scheduling

Feeds to Scour
SubscribedAll
Scoured 152 posts in 11.9 ms
GitHub·

I got tired of not understanding how vLLM works under the hood, so I built my own mini inference engine from scratch.

Discussed on r/LLM
Feeds
blog.skypilot.co·

SkyPilot Endpoints: Production-Ready Inference on Every Cluster You Own

Discussed on Hacker News
Feeds
blocksandfiles·

DDN launches faster array HW and KV Cache SW for AI

Feeds
lemmy.ml·

DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference

Feeds
Red Hat Developer·

Optimizing distributed AI inference: Advanced deployment patterns

Covers 3 stories See all stories this covers including DeepSeek-V3 Technical Report
Feeds
thecybersidekick.beehiiv.com·

AI Inference at the Edge: Running Real-Time LLMs in Kubernetes Without a GPU Farm

Discussed on DEV
Feeds
arXiv·

Cache-Resident LLM Inference in GB-Scale Last-Level Caches

Feeds
primeintellect.ai·

RL at 1T Scale: prime-rl Performance Deep Dive

Covers 6 stories See all stories this covers including Kimi K2.7-Code: open-source coding model with better token efficiency
Feeds
medium.com
·

The Hidden Memory Problem Behind Fast LLM Inference

Feeds
IBM Research·

Running AI on mixed hardware for speed and affordability

Covers Introduction to llm-d Open-source Kubernetes-native Framework for Distributed LLM Inference | Ep 140 #cloudnativefm
Feeds
Anyscale blog posts·

High Performance Distributed Inference with Ray Serve LLM

Covered by Google Cloud Blog
Discussed on Hacker News
Feeds
medium.com
·

Inside TurboQuant: The Algorithmic Breakthrough Smashing LLM Memory Walls

Feeds
supercomputing-system-ai-lab.github.io·

VoltanaLLM: Energy-Efficient LLM Serving

Discussed on Hacker News
Feeds
medium.com
·

PolyKV: We Gave 15 AI Agents One Shared Memory and It Actually Worked

Feeds
fitservers.com·

The Complete Guide to Deploying DeepSeek R1 on a Dedicated Server

Feeds
NVIDIA Technical Blog·

Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding

Covers 3 stories See all stories this covers including NVIDIA/TensorRT-LLM
Feeds
portal.neuralwatt.com·

Neuralwatt: Energy-based pricing for AI inference. Efficient prompts cost less

Discussed on Hacker News
Feeds
Hugging Face·

Qwen-AgentWorld-35B-A3B for Coding?

Discussed on r/LocalLLaMA
Feeds
digitalocean.com·

Multi-Model API Cost Governance with the Inference Router

Covers GitHub - vllm-project/semantic-router: Intelligent Router for Mixture-of-Models
Feeds
hamanlp.org·

Lean Zig by building an LLM from scratch

Covers Zig Software Foundation ⚡ Zig Programming Language
Discussed on Hacker News
Feeds

Keyboard Shortcuts

Navigation

Next / previous post
j/k
Open post
oorEnter
Preview post
v

Post Actions

Love post
a
Like post
l
Dislike post
d
Undo reaction
u
Save / unsave
s

Recommendations

Add interest / feed
Enter
Not interested
x

Go to

Home
gh
Interests
gi
Feeds
gf
Likes
gl
History
gy
Changelog
gc
Settings
gs
Discover
gb
Search
/

General

Show this help
?
Submit feedback
!
Close modal / unfocus
Esc

Press ? anytime to show this help