Adaptive speculative decoding: picking draft lengths at runtime

A follow-on to the economics of speculative decoding, we run the inference lab simulator on MTP & DFlash drafters with real acceptance data, and find out whether adaptively choosing the draft length is worth it.

View Post on Scour ↗

More from Fergus's blog on Scour.

Covers 4 stories including Looking for a self-hosted alternative to Modal.com for running ML workloads