Broken on Arrival: Non-Functional Quantized LLM Artifacts in Public Model Registries and How to Catch Them

Thanks for the note. The screen is built to catch files that are broken outright. The seven we found solve at most 6 of 164 tasks while still streaming text at normal speed, and the 15-task suite flags all of them, at a median of 30 seconds of generation per file on one consumer GPU. Schema-following and long-context recall need their own tests.

On whether tool use breaks first, a recent GGUF study of agentic tool use in Polish (arXiv 2609.32042, three 7B to 11B models) found no steady decline in task pass rates from Q8_0 to Q3_K_M and a collapse at Q2_K in all three, with format errors already present at Q8_0.

Any per-file suite also has to record the chat template it ran with. Two community deepseek-coder files in our sample carry no chat template. They scored 5 and 6 of 15 under the runtime's default ChatML and 13 and 15 with the model's own instruction template.

On the registry side, the paper recommends testing every low-bit tag before publication and reporting the result alongside its quantization recipe.

Probably, though we ran the screen on CUDA and Metal and have no CPU timing. It goes through Ollama, and an outside reader reproduced one tag's failure on a CPU-only Debian machine with a three-task check of their own (ollama issue 18252). Generation time scales with your tokens per second. At 20, Qwen2.5-Coder-3B takes about 2 minutes at q4_K_M (about 2,700 tokens across the 15 tasks) and about 10 at q2_K (11,520, since it hits the 768-token cap on every task), plus prompt processing, model load, download and scoring. Without download and scoring, the median across all tags was about 30 seconds on one RTX 4090, and the four failing 3B tags took about 2 to 6 minutes on an Apple M2 with Metal. The screen pulls the tag to test it, then removes the model unless you pass --keep-model.

Every functional test we ran was Python code: HumanEval+ in the census and MBPP+ in the earlier tuning runs. So we have no instruction-following or chat numbers. The one non-code measure is WikiText-2 perplexity. The five non-functional builds we measured sit at 27 to 5,700 times their fp16 model, while the two q4_K_M tags we measured stay within about 5%. Perplexity is not a functional verdict, but the damage shows on plain English text too.

For casual use we can only go by the code prompts, where it depends on the tag. Qwen-3B q2_K writes word salad up to the token cap, which is hard to miss. q3_K_M is harder to spot. On our CUDA run most answers are a signature, often a docstring, and no body, with stray periods and broken lines. That can pass a quick glance but not a careful read.

None of the five rebuilt files was corrupted, as you say. But the same uncalibrated recipe leaves the smaller Qwen2.5-Coder-1.5B functional, so the recipe alone does not predict failure. That is why the paper recommends testing every low-bit tag.