GGUF vs GPTQ vs AWQ: The Plain-English Guide to LLM Quantization (and Which One to Pick) (opens in new tab)

Covers GitHub here . You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inferen...Discussed on Hacker News

GGUF, GPTQ, AWQ, Q4_K_M, NF4 — the quantization alphabet soup, explained for people who just want to fit a bigger model in the VRAM they have. What each format is, the real VRAM math, and a decision table for which to use.

Read the original article