Huawei Open-Sourced MACaQ, Which Quantized a 122B Model in 64 Minutes on 32.5 GB. Here Is the Weekend Test I'd Run on One GPU.
Huawei's Computing Systems Lab in Switzerland open-sourced MACaQ (Model Agnostic Calibrated Quantization) under Apache 2.0, with a paper submitted to arXiv on November 7. The headline claim is that it can "quantize a 122B-parameter model in 64 minutes using just 32.5 GB of device memory." It outputs 2, 3 and 4-bit GGUF files for llama.cpp and W4A4 or weight-only checkpoints for vLLM. For anyone running local models, the question is not whether you need a 122B model. It is whether you can now quantize your own fine-tune on the GPU you already have.
What the claim says, and what it leaves out
The paper's abstract says the 122B run used "only 32.5GB GPU RAM." So this was a GPU run, and 32.5 GB fits on a single 48 GB workstation card. (It's just over the 32 GB on a consumer flagship, so read the number carefully.) That is the part worth noticing, if it holds up. Most of us download someone else's GGUF instead of making our own, partly because the tooling and hardware for doing it well feel out of reach.
What's missing matters too. Neither the README nor the abstract names the 122B model, and neither names the GPU. A 64-minute figure on an unnamed card is a hard number to plan around. I'd treat it as "a large model on one card in about an hour" and verify it on my own hardware.
On quality, the authors compare against MR-GPTQ and DuQuant++ and claim lower perplexity, or the same best Llama-3-8B perplexity "with 11× less quantization time." That's a comparison against research methods, not against what most of us use. I found no head-to-head with llama.cpp's own importance-matrix quants or with AWQ in the material I read, and that is the comparison I care about.
The numbers I can check
The README has one concrete table I can use. On Qwen3.8-27B served through vLLM, BF16 perplexity is 2.8463 and MACaQ W4A4 is 2.8994. Prefill runs 2.33x faster and decode 2.58x faster than BF16. The lab's blog reports a 2.74x decode speedup in llama.cpp on the same model versus FP16.
A perplexity rise from 2.85 to 2.90 is small. Whether it's small for your task is a different question. Perplexity averages over everything, and quantization damage tends to show up first in the long tail: tool calls with exact JSON, code, and rare vocabulary. Those are the things my products depend on.
What it supports
The text side is mostly the Qwen family: Qwen3-4B, Qwen3.8-27B, Qwen3-30B-A3B, Qwen3-Coder-30B-A3B-Instruct, Qwen3.6-35B-A3B and Qwen3-Next-80B-A3B-Instruct, plus AllenAI's OLMoE. It also covers image and video models (FLUX.2 klein 4B and Wan2.2 TI2V 5B) and speech (Qwen3-TTS and Chatterbox Turbo). The llama.cpp path also supports CPU execution, which matters if you're on a Mac.
There's also Ascend NPU kernel work in CANN pull requests, but the README says full-model execution "requires integration with an Ascend inference runtime." Unless you run Huawei hardware, ignore that part.
The weekend test I'd run
I wouldn't touch production for this. I would spend a Saturday morning on one question: does a MACaQ 4-bit GGUF of a model I already use beat the Q4 GGUF I already have?
Pick one supported model you already run locally. For me that's Qwen3-30B-A3B, because I use it for extraction jobs. Download the existing community Q4 GGUF you use today as the baseline. Quantize the same base model with MACaQ to a 4-bit GGUF, timing the run and watching peak GPU memory. Then run your own eval, not perplexity. I keep 50 real prompts from my products with known-good outputs, scored by exact match on the JSON fields. Compare tokens per second in llama.cpp on the same machine.
If MACaQ's file scores the same or better on my 50 prompts and runs at least as fast, I'd switch my quantization step to it, mainly because it lets me quantize my own fine-tunes locally. If it's within noise, I'd stay with what I have. A repo with 16 commits when I looked is not a reason to rebuild a pipeline that works.
The honest take
The interesting thing about MACaQ is not the 122B number. It's that calibrated low-bit quantization is getting cheap enough to run on hardware a solo builder owns, under a license that lets you use it commercially. If you fine-tune small models for your product, that changes your workflow: train, quantize, and test on one box, with no rented GPU in the loop.
Where I could be wrong: this is a week-old research release with an unnamed headline model and no comparison against the quants most people already use. It may turn out that llama.cpp's own quants are just as good for typical models, and MACaQ's edge only shows at W4A4 on vLLM. That's why the test above is a Saturday morning, not a migration.
Author
Lukas
@lukcombinator