💻 Local LLMs - nmarshall

🧠AI News Blog

blog.google··Hacker News

zhongkaifu/TensorSharp: A C# inference engine for running large language models (LLMs) locally using GGUF model files. TensorSharp provides a console application, a web-based chatbot interface, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/Linux with full GPU capability

🧠AI Code

github.com··Hacker News

Show HN: Run Llama.cpp In-Process from Java with Project Panama FFM

🤖AI agents

deemwar-products.github.io··Hacker News

Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression

🌱Gardening Academic

arxiv.org·

Unsloth Gemma 4 QAT

🏠Self-hosted AI

unsloth.ai·

harshuljain13/llm-inference-at-scale: A Practitioner handbook for production llm serving.

🤖AI Inference Code

github.com··Hacker News

MoQ GGUFs and GSQ: Low-Bit GGUFs Are About to Get Much Better

🏠Self-hosted AI News Blog

kaitchup.substack.com··r/LocalLLaMA

local llm on laptop 780M GPU using llama + gemma 4 qat

🏠Self-hosted AI Blog

alper.bearblog.dev·

Using Scikit-LLM with Open-Source LLMs

🧠AI

machinelearningmastery.com·

Running LLM Inference on Kubernetes: What It Actually Takes

🧠AI Blog

fairwinds.com·

huawei-csl/KVarN: KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

🤖AI Inference Code

github.com··Hacker News

LLM Inference Engineering Room — Part 3: The Orchestration Layer

🤖AI Inference Blog

vimal-dwarampudi.medium.com·

stable-diffusion.cpp/docs/quantization_and_gguf.md at master · leejet/stable-diffusion.cpp

🏠Self-hosted AI Code

github.com··r/StableDiffusion

Nvidia DGX Spark GB10 – AI Models and Guide with vLLM and Autonomous Script

🏗️AI Infrastructure Code

github.com··Hacker News

Value-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Models

📱Edge AI Academic

arxiv.org·

Improved performance and model support with GGUF

UniSVQ: 2-bit Unified Scalar-Vector Quantization

Ollama 0.30 delivers faster NVIDIA GPU performance and wider hardware support

Qwen 3.6 27B AutoRound GGUF, need your feedback

GGUF vs GPTQ vs AWQ: The Plain-English Guide to LLM Quantization (and Which One to Pick)

Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency

Show HN: Run Llama.cpp In-Process from Java with Project Panama FFM

Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression

Unsloth Gemma 4 QAT

harshuljain13/llm-inference-at-scale: A Practitioner handbook for production llm serving.

MoQ GGUFs and GSQ: Low-Bit GGUFs Are About to Get Much Better

local llm on laptop 780M GPU using llama + gemma 4 qat

Using Scikit-LLM with Open-Source LLMs

Running LLM Inference on Kubernetes: What It Actually Takes

huawei-csl/KVarN: KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

LLM Inference Engineering Room — Part 3: The Orchestration Layer

stable-diffusion.cpp/docs/quantization_and_gguf.md at master · leejet/stable-diffusion.cpp

Nvidia DGX Spark GB10 – AI Models and Guide with vLLM and Autonomous Script

Value-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Models