I built a Rust entropy monitor to route LLM inference — here's what the benchmark showed (opens in new tab)

Frontier LLM inference is expensive. I wanted to see how far a 4B local model could go before needing a cloud call — and when the cloud call actually adds value. The result is Buddy System: a tiered inference architecture where a Rust entropy monitor watches per-token uncertainty during local generation and routes to Sonnet only when the local model is genuinely stuck. (I know Anthropic has the advisor system, but this is different) GitHub: How it works Gemma 3 4B generates locally on Apple S...