🤖 AI Inference - nmarshall

💻Local LLMs Academic

arxiv.org·

Running LLM Inference on Kubernetes: What It Actually Takes

🧠AI Blog

fairwinds.com·

Token4Token — pay-per-token inference on Gnosis + Swarm

🧠AI

t4t.eth.link··Hacker News

KJLdefeated/RL.cu: RLVR training for LLM in CUDA/C++

🔥PyTorch Code

github.com··Hacker News

LLM Inference Engineering Room — Part 3: The Orchestration Layer

💻Local LLMs Blog

vimal-dwarampudi.medium.com·

Build a Medical Report Analyzer on Dedicated Inference with Python

💻Local LLMs

digitalocean.com·

Speculators v0.5.0: DFlash support and online training

🏗️AI Infrastructure

developers.redhat.com·

APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing

🔥Burn Academic

arxiv.org·

Nvidia DGX Spark GB10 – AI Models and Guide with vLLM and Autonomous Script

🏗️AI Infrastructure Code

github.com··Hacker News

ju4nv1e1r4/nlp_engine_inference: An inference engine for NLP models.

📝NLP Code

github.com··r/rust

huawei-csl/KVarN: KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

💻Local LLMs Code

github.com··Hacker News

heterodoxin/graphkv: Graph-guided KV cache compression for memory-efficient LLM inference.

💻Local LLMs Code

github.com··r/LocalLLaMA

zhongkaifu/TensorSharp: A C# inference engine for running large language models (LLMs) locally using GGUF model files. TensorSharp provides a console application, a web-based chatbot interface, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/Linux with full GPU capability

💻Local LLMs Code

github.com··Hacker News

fix(gateway): fail closed for unknown model auth · openclaw/openclaw@85343ea

🏗️AI Infrastructure Code

github.com·

FlexNPU: Transparent NPU Virtualization for Dynamic LLM Prefill-Decode Co-location

⚡Hardware Acceleration Academic

arxiv.org·

mirkolenz/llmhop: Tiny, stateless Go router that dispatches OpenAI-compatible requests to single-model vLLM and sglang backends with zero external dependencies

🧠AI Code

github.com··Hacker News

harshuljain13/llm-inference-at-scale: A Practitioner handbook for production llm serving.

DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time - Huawei, GB300 NVL72, MI355X, B200

Infrastructure Options for Scalable AI Inference

Inferoa AI harness claimed 90% cache savings. We ran it and measured 97.8%

Breaking the Ice: Analyzing Cold Start Latency in vLLM

Running LLM Inference on Kubernetes: What It Actually Takes

Token4Token — pay-per-token inference on Gnosis + Swarm

KJLdefeated/RL.cu: RLVR training for LLM in CUDA/C++

LLM Inference Engineering Room — Part 3: The Orchestration Layer

Build a Medical Report Analyzer on Dedicated Inference with Python

Speculators v0.5.0: DFlash support and online training

APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing

Nvidia DGX Spark GB10 – AI Models and Guide with vLLM and Autonomous Script

ju4nv1e1r4/nlp_engine_inference: An inference engine for NLP models.

huawei-csl/KVarN: KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

heterodoxin/graphkv: Graph-guided KV cache compression for memory-efficient LLM inference.

fix(gateway): fail closed for unknown model auth · openclaw/openclaw@85343ea

FlexNPU: Transparent NPU Virtualization for Dynamic LLM Prefill-Decode Co-location

mirkolenz/llmhop: Tiny, stateless Go router that dispatches OpenAI-compatible requests to single-model vLLM and sglang backends with zero external dependencies