# Push 4-bit converted model to hub

**URL:** <https://discuss.huggingface.co/t/push-4-bit-converted-model-to-hub/59967>\
**Category:** Models\
**Created:** [October 26, 2023, 7:40am UTC](https://discuss.huggingface.co/t/push-4-bit-converted-model-to-hub/59967 "2023-10-26T07:40:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ckandemir](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ckandemir/32/22679_2.png) [@ckandemir](https://discuss.huggingface.co/u/ckandemir)\
**Post date:** [October 26, 2023, 7:40am UTC](https://discuss.huggingface.co/t/push-4-bit-converted-model-to-hub/59967/1 "2023-10-26T07:40:57Z")

</div>

Hey there,

I am quantizing a model to 4-bit using `BitsandBytes`, and when I try to push the model to the hub I am getting the following error:

```auto
You are calling `save_pretrained` on a 4-bit converted model. This is currently not supported

```

I’ve pushed a [4-bit converted model](https://huggingface.co/ckandemir/codellama-4bit) to the hub before after finetuning it with peft, however I was wondering whether I can do it without going through that path.

I know `GPTQ` gives me more options, but it only works for text models.

Alternatively I can go through the finetuning and use methods like pruning, distillation to get the model smaller, but I am wondering if there is work around for pushing a 4-bit converted model to the hub

---

<div class="post-metadata">

**Author:** ![ckandemir](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ckandemir/32/22679_2.png) [@ckandemir](https://discuss.huggingface.co/u/ckandemir)\
**Post date:** [October 27, 2023, 7:28am UTC](https://discuss.huggingface.co/t/push-4-bit-converted-model-to-hub/59967/2 "2023-10-27T07:28:47Z")

</div>

Following PRs do handle this issue^

> <https://github.com/TimDettmers/bitsandbytes/pull/753>
>
> Purpose: enable saving and loading transformers models in 4bit formats.
> Enables… this PR in transformers: https://github.com/huggingface/transformers/pull/26037
> 
> addresses feature request #603 and other similar ones elsewhere.
> 
> 
> tested with Bloom-560 and Llama-2-7b. 
> tested quantization, saving, loading, -- matched tensors and quant\_states -- matched inference results.
> tested both nf4 and fp4
> test added

> <https://github.com/huggingface/transformers/pull/26037>
>
> \## What does this PR do?
> Purpose: enable saving and loading transformers models… in 4bit formats.
> 
> tested with Bloom-560 and Llama-2-7b. Save - Load - match tensors and quant\_states - match inference results.
> 
> \## connection with bitsandbytes
> Requires this PR in bitsandbytes: https://github.com/TimDettmers/bitsandbytes/pull/753 to be able to save/load models
> 
> \## testing:
> the functionality was tested doing this series of commands:
> \`\`\`
> model = m4 = transformers.AutoModelForCausalLM.from\_pretrained(..., \[with 4bit quant config\])
> model.save\_pretrained(SAVE\_PATH, safe\_serialization= \[False / True\] )
> model2 = transformers.AutoModelForCausalLM.from\_pretrained( SAVE\_PATH)
> \# then matching all params and quant\_state items between the models, matching inference results
> \`\`\`
> Specific tests will be added to this PR once the \[bitsandbytes PR\](https://github.com/TimDettmers/bitsandbytes/pull/753) merges.
> 
> \## Open questions to the maintainers:
> 1. I added and tested code necessary for straightforward save/load operations with LLMs. Yet there may be other kinds of function calls that may need to be updated to handle 4bit save/load - pls suggest if/where to expand this PR.
> specifically: this PR covers \`load\_state\_dict\_into\_meta\_model()\` but not \`load\_state\_dict\_into\_model()\` for I did not find an example that uses it. 
> 
> 2. Some of my edits may not fit with style / refactoring plans of the maintainers - pls give guidance if needed.

I will just wait until they are merged 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [February 18, 2024, 10:42am UTC](https://discuss.huggingface.co/t/push-4-bit-converted-model-to-hub/59967/3 "2024-02-18T10:42:08Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.