💻 Local LLMs - matmat

📊Quantization News Blog

blog.google··Hacker News

google/gemma-4-12B-it-qat-q4_0-gguf

🎓Academic Torrents

huggingface.co·

Using Scikit-LLM with Open-Source LLMs

🧪Data science

machinelearningmastery.com·

Show HN: Run Llama.cpp In-Process from Java with Project Panama FFM

🐚Shell Automation

deemwar-products.github.io··Hacker News

Apples to Apples: MLX vs. Llama.cpp for Gemma 4 12B on an M1 16GB

🎯Emulator Accuracy Blog

ziraph.com··Hacker News

Google AI Edge Gallery launches to macOS

🎙️Whisper

9to5mac.com··r/apple

techjarves/Portable-AI-USB: A 100% offline, fully portable, zero-trace AI (Ollama + Llama 3 + AnythingLLM) that runs natively from a USB drive on Windows and Mac.

🔓Open Source Software Code

github.com·

Google Deepmind's Gemma 4 12B squeezes multimodal AI onto a laptop with just 16 GB of RAM

🔓Free and open source

the-decoder.com

Gemma 4 12B: A unified, encoder-free multimodal model

📄Text Chunking Discussion

news.ycombinator.com··Hacker News

zhongkaifu/TensorSharp: A C# inference engine for running large language models (LLMs) locally using GGUF model files. TensorSharp provides a console application, a web-based chatbot interface, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/Linux with full GPU capability

⚡Parallel Computing Code

github.com··Hacker News

Nvidia DGX Spark GB10 – AI Models and Guide with vLLM and Autonomous Script

⚡Parallel Computing Code

github.com··Hacker News

huawei-csl/KVarN: KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

⚡Parallel Computing Code

github.com··Hacker News

ulyssestenn/omt: Ollama Model Test - Figure out the best model for the task

🔤Character Classification Code

github.com··Hacker News

mirkolenz/llmhop: Tiny, stateless Go router that dispatches OpenAI-compatible requests to single-model vLLM and sglang backends with zero external dependencies

💾File Formats Code

github.com··Hacker News

zaydmulani09/mnemo: Local-first AI memory layer for any LLM. Persistent knowledge graph, entity extraction, semantic retrieval. Works with Ollama, OpenAI, Anthropic, or any OpenAI-compatible backend.

📝SQLite WAL Code

github.com··Hacker News

john-rocky/apple-silicon-llm-bench: Neutral, reproducible benchmark for local LLMs on Apple Silicon (Mac · iPhone · iPad) — MLX, llama.cpp, CoreML, Apple Foundation Models

💻Operating System, OS Code

github.com··Hacker News

No more posts from matmat's subscribed feeds.

Scour all 25255 feeds Learn more about Feeds

GGUF vs GPTQ vs AWQ: The Plain-English Guide to LLM Quantization (and Which One to Pick)

defai-digital/ax-engine: Apple Silicon LLM runtime supporting Gemma 4 and Qwen 3.6 MTP modes

Fixing a stuck Ollama runner and building a GPU watchdog

Unsloth Gemma 4 QAT

Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency

google/gemma-4-12B-it-qat-q4_0-gguf

Using Scikit-LLM with Open-Source LLMs

Show HN: Run Llama.cpp In-Process from Java with Project Panama FFM

Apples to Apples: MLX vs. Llama.cpp for Gemma 4 12B on an M1 16GB

Google AI Edge Gallery launches to macOS

techjarves/Portable-AI-USB: A 100% offline, fully portable, zero-trace AI (Ollama + Llama 3 + AnythingLLM) that runs natively from a USB drive on Windows and Mac.

Google Deepmind's Gemma 4 12B squeezes multimodal AI onto a laptop with just 16 GB of RAM

Gemma 4 12B: A unified, encoder-free multimodal model

Nvidia DGX Spark GB10 – AI Models and Guide with vLLM and Autonomous Script

huawei-csl/KVarN: KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

ulyssestenn/omt: Ollama Model Test - Figure out the best model for the task

mirkolenz/llmhop: Tiny, stateless Go router that dispatches OpenAI-compatible requests to single-model vLLM and sglang backends with zero external dependencies

zaydmulani09/mnemo: Local-first AI memory layer for any LLM. Persistent knowledge graph, entity extraction, semantic retrieval. Works with Ollama, OpenAI, Anthropic, or any OpenAI-compatible backend.

john-rocky/apple-silicon-llm-bench: Neutral, reproducible benchmark for local LLMs on Apple Silicon (Mac · iPhone · iPad) — MLX, llama.cpp, CoreML, Apple Foundation Models