# NotImplementedError: ggml\_type 21 not implemented

**URL:** <https://discuss.huggingface.co/t/notimplementederror-ggml-type-21-not-implemented/108104>\
**Category:** 🤗Transformers\
**Created:** [September 23, 2024, 9:55am UTC](https://discuss.huggingface.co/t/notimplementederror-ggml-type-21-not-implemented/108104 "2024-09-23T09:55:09Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![gfatigati](https://avatars.discourse-cdn.com/v4/letter/g/85e7bf/32.png) [@gfatigati](https://discuss.huggingface.co/u/gfatigati)\
**Post date:** [September 23, 2024, 9:55am UTC](https://discuss.huggingface.co/t/notimplementederror-ggml-type-21-not-implemented/108104/1 "2024-09-23T09:55:09Z")

</div>

I’m running a 3 bit quantized model of Llama 3.1 70b:

```auto
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "bartowski/Meta-Llama-3.1-70B-Instruct-GGUF"
filename = "Meta-Llama-3.1-70B-Instruct-IQ3_XS.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)

```

having the following error:

```auto

  File "***/lib/python3.11/site-packages/transformers/integrations/ggml.py", line 510, in load_dequant_gguf_tensor
    raise NotImplementedError(
NotImplementedError: ggml_type 21 not implemented

```

I’m using

- transformers 4.44.2
- gguf 0.10.0

I’m doing something wrong? Thanks.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [September 23, 2024, 11:34am UTC](https://discuss.huggingface.co/t/notimplementederror-ggml-type-21-not-implemented/108104/2 "2024-09-23T11:34:35Z")

</div>

There are multiple issues, but to put it simply, **the GGUF files that can be used in transformers and the GGUF files that can be used in Llamacpp are currently effectively two different things**.  
Think of it as two different dialects.  
I think GGUF was originally a proprietary format at the Llamacpp development…  
The one you are trying to use is for Llamacpp. The author of the quantization can usually tell.

And since it is unlikely that anyone publishes a GGUF for transformers, you can either use that GGUF in Llamacpp or find a non-GGUF quantization file (such as GPTQ or bitsandbytes NF4) for transformers.

---

<div class="post-metadata">

**Author:** ![gfatigati](https://avatars.discourse-cdn.com/v4/letter/g/85e7bf/32.png) [@gfatigati](https://discuss.huggingface.co/u/gfatigati)\
**Post date:** [September 23, 2024, 12:15pm UTC](https://discuss.huggingface.co/t/notimplementederror-ggml-type-21-not-implemented/108104/3 "2024-09-23T12:15:03Z")

</div>

Understood, thanks!