# Possible to use transformers with GGML-style quantization?

**URL:** <https://discuss.huggingface.co/t/possible-to-use-transformers-with-ggml-style-quantization/72648>\
**Category:** 🤗Transformers\
**Created:** [February 10, 2024, 2:50pm UTC](https://discuss.huggingface.co/t/possible-to-use-transformers-with-ggml-style-quantization/72648 "2024-02-10T14:50:13Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![neilf79](https://avatars.discourse-cdn.com/v4/letter/n/cc9497/32.png) [@neilf79](https://discuss.huggingface.co/u/neilf79)\
**Post date:** [February 10, 2024, 2:50pm UTC](https://discuss.huggingface.co/t/possible-to-use-transformers-with-ggml-style-quantization/72648/1 "2024-02-10T14:50:13Z")

</div>

Hi!

Is it possible to use the transformers library with GGML-style quantization? For example, GGML\_TYPE\_Q5\_K (as detailed [here](https://huggingface.co/TheBloke/CodeLlama-7B-GGUF#explanation-of-quantisation-methods)).

I’m aware that I can use bitsandbytes to do 8-bit and 4-bit quantization, but I’d prefer to use GGML-style quantization.

Thanks!