⚡ LLM Quantization - akapaka

Less-relevant results

KaiFelixBennett/gemma4-turboquant-rdna4: Run Gemma-4-31B at full 256K context on a $1,400 AMD RDNA4 GPU (gfx1201): TurboQuant KV cache + HIP-graph-safe Flash-Attention for llama.cpp, fully measured on real hardware.

🧠LLM Inference Code

github.com··Hacker News

MoQ GGUFs and GSQ: Low-Bit GGUFs Are About to Get Much Better

🧠LLM Inference News Blog

kaitchup.substack.com··r/LocalLLaMA

Here's a llama.cpp CLI Command builder.

🧠Local llm

llamabuilding.com··r/LocalLLaMA

DeskDash - a free Windows tool to easily manage your GGUF files

🧠Local llm

gerry7.itch.io··r/LocalLLaMA

Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency

🧠Local llm News Blog

blog.google··Hacker News

Evaluating bigaspv2-5, a Flow Matching Alternative to SDXL

🧠Local llm

hackernoon.com·

Show HN: Run Llama.cpp In-Process from Java with Project Panama FFM

🧠Local llm

deemwar-products.github.io··Hacker News

mtmd : add video input support by ngxson · Pull Request #24269 · ggml-org/llama.cpp

🧠Local llm Code

github.com··r/LocalLLaMA

Apples to Apples: MLX vs. Llama.cpp for Gemma 4 12B on an M1 16GB

🧠Local llm Blog

ziraph.com··Hacker News

Apple WWDC On-Device AI Deep Dive - Google Docs

🧠LLM Inference

gist.is··Hacker News

Running Qwen 35B MoE at 450k Context on a Single 32GB GPU

🧠LLM Inference

local-llm.utop.workers.dev··Hacker News

mtp: support for gemma-4 E2B and E4B assistants by max-krasnyansky · Pull Request #24282 · ggml-org/llama.cpp

🧠Local llm Code

github.com··r/LocalLLaMA

A system programmer’s guide to LLM inference

🧠LLM Inference Blog

blog.xiangpeng.systems··Hacker News

Remove padding and multiple D2D copies for MTP by gaugarg-nv · Pull Request #24086 · ggml-org/llama.cpp

🧠Local llm Code

github.com··r/LocalLLaMA

Launch HN: General Instinct (YC P26) – Frontier models on edge devices

🧠Local llm Discussion

news.ycombinator.com··Hacker News

1-bit and 1.58 bit LLM Benchmarking on Jetson Orin Nano Super | Bonsai LM

🤖Qwen

smolhub.com··r/LocalLLaMA

bigattichouse/packed-twin-inference: PTI achieves ~2× throughput using a single quantized model (Q5_K_M or better) by running 4 generation streams in one batched decode call. The GPU loads model weights once per step and produces 4 predictions simultaneously. KV cache overhead is ~0.8 GiB total for all 4 streams. No draft model. No quality loss

🧠LLM Inference Code

github.com··r/LocalLLaMA

john-rocky/apple-silicon-llm-bench: Neutral, reproducible benchmark for local LLMs on Apple Silicon (Mac · iPhone · iPad) — MLX, llama.cpp, CoreML, Apple Foundation Models

🤖Qwen Code

github.com··Hacker News

GGUF vs GPTQ vs AWQ: The Plain-English Guide to LLM Quantization (and Which One to Pick)

Qwen 3.6 27B AutoRound GGUF, need your feedback

KaiFelixBennett/gemma4-turboquant-rdna4: Run Gemma-4-31B at full 256K context on a $1,400 AMD RDNA4 GPU (gfx1201): TurboQuant KV cache + HIP-graph-safe Flash-Attention for llama.cpp, fully measured on real hardware.

MoQ GGUFs and GSQ: Low-Bit GGUFs Are About to Get Much Better

Here's a llama.cpp CLI Command builder.

DeskDash - a free Windows tool to easily manage your GGUF files

Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency

Evaluating bigaspv2-5, a Flow Matching Alternative to SDXL

Show HN: Run Llama.cpp In-Process from Java with Project Panama FFM

mtmd : add video input support by ngxson · Pull Request #24269 · ggml-org/llama.cpp

Apples to Apples: MLX vs. Llama.cpp for Gemma 4 12B on an M1 16GB

Apple WWDC On-Device AI Deep Dive - Google Docs

Running Qwen 35B MoE at 450k Context on a Single 32GB GPU

mtp: support for gemma-4 E2B and E4B assistants by max-krasnyansky · Pull Request #24282 · ggml-org/llama.cpp

A system programmer’s guide to LLM inference

Remove padding and multiple D2D copies for MTP by gaugarg-nv · Pull Request #24086 · ggml-org/llama.cpp

Launch HN: General Instinct (YC P26) – Frontier models on edge devices

1-bit and 1.58 bit LLM Benchmarking on Jetson Orin Nano Super | Bonsai LM

john-rocky/apple-silicon-llm-bench: Neutral, reproducible benchmark for local LLMs on Apple Silicon (Mac · iPhone · iPad) — MLX, llama.cpp, CoreML, Apple Foundation Models