# Quantizing a model on M1 Mac for qlora

**URL:** <https://discuss.huggingface.co/t/quantizing-a-model-on-m1-mac-for-qlora/77356>\
**Category:** 🤗Transformers\
**Created:** [March 14, 2024, 4:21am UTC](https://discuss.huggingface.co/t/quantizing-a-model-on-m1-mac-for-qlora/77356 "2024-03-14T04:21:11Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jkunstle](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@jkunstle](https://discuss.huggingface.co/u/jkunstle)\
**Post date:** [March 14, 2024, 4:21am UTC](https://discuss.huggingface.co/t/quantizing-a-model-on-m1-mac-for-qlora/77356/1 "2024-03-14T04:21:11Z")

</div>

This may be more of a PEFT question than a Transformers question-

I’d like to fine-tune a Mistral-7b model on my 32GB M1 Pro MacBook. I’ve found support to do this with MLX in 4-bit qlora but I’d like to stay in the HF + torch ecosystem if possible. In BF16 my machine spends more time handling swap than using the GPU.

BitsandBytes supports 4-bit quantization for models but those layers aren’t supported for non-cuda backends.

Any recommendations on how this could be done? Thanks!