🤖 AI Inference - nmarshall

⚡Performance News Blog

blog.google··Hacker News, r/LocalLLaMA, r/singularity

How we fight GPU scarcity without compromise

✍️Prompt Engineering Blog

equixly.com··Hacker News

defai-digital/ax-engine: Apple Silicon LLM runtime supporting Gemma 4 and Qwen 3.6 MTP modes

🤖AI agents Code

github.com··Hacker News

Create Your Own Programming Language with Rust

🔨Compilers

createlang.rs··Hacker News

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

🏗️AI Infrastructure Academic

arxiv.org··Hacker News

Your Lambda isn't leaking memory — your metrics are lying to you

☁️Cloud Computing Blog

engineering.taktile.com··Hacker News

OpenEnv is now owned by HF, Torch, Prime Intellect, Unsloth, Modal, Mercor, and more! Use it for training agents.

🤖Reinforcement Learning Blog

huggingface.co··Hacker News, r/LocalLLaMA

KJLdefeated/RL.cu: RLVR training for LLM in CUDA/C++

🔥PyTorch Code

github.com··Hacker News

Running Qwen 35B MoE at 450k Context on a Single 32GB GPU

💻Local LLMs

local-llm.utop.workers.dev··Hacker News

Nvidia DGX Spark GB10 – AI Models and Guide with vLLM and Autonomous Script

🏗️AI Infrastructure Code

github.com··Hacker News

GGUF vs GPTQ vs AWQ: The Plain-English Guide to LLM Quantization (and Which One to Pick)

🏠Self-hosted AI

vettedconsumer.com··Hacker News

Nvidia's RTX Spark is a developer's dream, but AMD's Ryzen AI Max+ is what most people actually need for local AI

🖥computers

xda-developers.com·

zhongkaifu/TensorSharp: A C# inference engine for running large language models (LLMs) locally using GGUF model files. TensorSharp provides a console application, a web-based chatbot interface, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/Linux with full GPU capability

💻Local LLMs Code

github.com··Hacker News

Nvidia enters PC chip market

🖥computers

jonpeddie.com·

mirkolenz/llmhop: Tiny, stateless Go router that dispatches OpenAI-compatible requests to single-model vLLM and sglang backends with zero external dependencies

🧠AI Code

github.com··Hacker News

Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency

💻Local LLMs News Blog

blog.google··Hacker News

No more posts from nmarshall's subscribed feeds.

Scour all 25255 feeds Learn more about Feeds

harshuljain13/llm-inference-at-scale: A Practitioner handbook for production llm serving.

DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time - Huawei, GB300 NVL72, MI355X, B200

Inferoa AI harness claimed 90% cache savings. We ran it and measured 97.8%

Token4Token — pay-per-token inference on Gnosis + Swarm

DiffusionGemma: 4x Faster Text Generation

How we fight GPU scarcity without compromise

defai-digital/ax-engine: Apple Silicon LLM runtime supporting Gemma 4 and Qwen 3.6 MTP modes

Create Your Own Programming Language with Rust

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

Your Lambda isn't leaking memory — your metrics are lying to you

OpenEnv is now owned by HF, Torch, Prime Intellect, Unsloth, Modal, Mercor, and more! Use it for training agents.

KJLdefeated/RL.cu: RLVR training for LLM in CUDA/C++

Running Qwen 35B MoE at 450k Context on a Single 32GB GPU

Nvidia DGX Spark GB10 – AI Models and Guide with vLLM and Autonomous Script

GGUF vs GPTQ vs AWQ: The Plain-English Guide to LLM Quantization (and Which One to Pick)

Nvidia's RTX Spark is a developer's dream, but AMD's Ryzen AI Max+ is what most people actually need for local AI

Nvidia enters PC chip market

mirkolenz/llmhop: Tiny, stateless Go router that dispatches OpenAI-compatible requests to single-model vLLM and sglang backends with zero external dependencies

Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency