# valueError: Supplied state dict for layers does not contain \`bitsandbytes\_\_\*\` and possibly other \`quantized\_stats\`(when load saved quantized model)

**URL:** <https://discuss.huggingface.co/t/valueerror-supplied-state-dict-for-layers-does-not-contain-bitsandbytes-and-possibly-other-quantized-stats-when-load-saved-quantized-model/110808>\
**Category:** 🤗Transformers\
**Created:** [October 9, 2024, 2:54am UTC](https://discuss.huggingface.co/t/valueerror-supplied-state-dict-for-layers-does-not-contain-bitsandbytes-and-possibly-other-quantized-stats-when-load-saved-quantized-model/110808 "2024-10-09T02:54:52Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![wichofer](https://avatars.discourse-cdn.com/v4/letter/w/d26b3c/32.png) [@wichofer](https://discuss.huggingface.co/u/wichofer)\
**Post date:** [October 9, 2024, 2:54am UTC](https://discuss.huggingface.co/t/valueerror-supplied-state-dict-for-layers-does-not-contain-bitsandbytes-and-possibly-other-quantized-stats-when-load-saved-quantized-model/110808/1 "2024-10-09T02:54:52Z")

</div>

We are trying to deploy a quantized Llama 3.1 70B model(from Huggingface, using bitsandbytes), quantizing part works fine as we check the model memory which is correct and also test getting predictions for the model, which is also correct, the problem is: after saving the quantized model and then loading it we get

> valueError: Supplied state dict for layers.0.mlp.down\_proj.weight does not contain `bitsandbytes__*` and possibly other `quantized_stats` components

What we do is:

- Save the quantized model using the usual save\_pretrained(save\_dir)
- Try to load the model using AutoModel.from\_pretrained, passing the save\_dir and the same quantization\_config used when creating the model.

Here is the code:

```auto
model_id = "meta-llama/Meta-Llama-3.1-70B-Instruct"

cache_dir = "/home/ec2-user/SageMaker/huggingface_cache"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model_4bit = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    quantization_config=quantization_config,
    low_cpu_mem_usage=True,
    offload_folder="offload",
    offload_state_dict=True,
    cache_dir=cache_dir
)

tokenizer = AutoTokenizer.from_pretrained(model_id,cache_dir=cache_dir)

pt_save_directory = "test_directory"
tokenizer.save_pretrained(pt_save_directory,)
model_4bit.save_pretrained(pt_save_directory)
## test load it

loaded_model = AutoModel.from_pretrained(pt_save_directory,
                                     quantization_config=quantization_config
                                     )

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 10, 2024, 10:01am UTC](https://discuss.huggingface.co/t/valueerror-supplied-state-dict-for-layers-does-not-contain-bitsandbytes-and-possibly-other-quantized-stats-when-load-saved-quantized-model/110808/2 "2024-10-10T10:01:48Z")

</div>

typo.

```py
tokenizer.save_pretrained(pt_save_directory,)

```

to

```py
tokenizer.save_pretrained(pt_save_directory)

```

---

<div class="post-metadata">

**Author:** ![schnell18](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@schnell18](https://discuss.huggingface.co/u/schnell18)\
**Post date:** [October 22, 2024, 12:10pm UTC](https://discuss.huggingface.co/t/valueerror-supplied-state-dict-for-layers-does-not-contain-bitsandbytes-and-possibly-other-quantized-stats-when-load-saved-quantized-model/110808/3 "2024-10-22T12:10:21Z")

</div>

This is a bug in the `_load_pretrained_model()` function of  
`transformers/modeling_utils.py` when loading sharded weight files. The  
state\_dict is applied to the empty model per shard. This is problematic as the  
quantized weight and its meta data(`*.quant_state.bitsandbytes__nf4`) may be  
stored in the different shards. The [quick-and-dirty fix](https://github.com/schnell18/transformers/commit/57d3073d1830520891c00ead77a0ac188be8bf49) is to merge tensors  
from all shards into one state\_dict. Similar issues have been reported on [stackoverflow](https://stackoverflow.com/questions/79068298/valueerror-supplied-state-dict-for-layers-does-not-contain-bitsandbytes-an)  
and the unsloth github issue 638

---

<div class="post-metadata">

**Author:** ![ZainZia](https://avatars.discourse-cdn.com/v4/letter/z/ccd318/32.png) [@ZainZia](https://discuss.huggingface.co/u/ZainZia)\
**Post date:** [May 30, 2025, 3:52pm UTC](https://discuss.huggingface.co/t/valueerror-supplied-state-dict-for-layers-does-not-contain-bitsandbytes-and-possibly-other-quantized-stats-when-load-saved-quantized-model/110808/4 "2025-05-30T15:52:01Z")

</div>

Is this issue fixed ?

I am getting the same error

in

import torch  
from transformers import (  
pipeline,  
BitsAndBytesConfig,  
)

# 1) Build your 4-bit config.

bnb\_config = BitsAndBytesConfig(  
load\_in\_4bit=True,  
llm\_int8\_enable\_fp32\_cpu\_offload=True, # keeps some weights in FP32 on CPU  
bnb\_4bit\_quant\_type=“nf4”, # or “fp4”, “fp4-dq”, etc.  
bnb\_4bit\_compute\_dtype=torch.float16, # compute in fp16 on GPU  
)

# 2) Create the pipeline, passing quantization\_config:

pipe = pipeline(  
“image-text-to-text”,  
model=“unsloth/gemma-3-27b-it-unsloth-bnb-4bit”,  
cache\_dir=“/mnt/models/gemma3”,  
trust\_remote\_code=True,  
device\_map=“auto”,  
quantization\_config=bnb\_config, # ← here’s the key  
)

messages = [  
{  
“role”: “user”,  
“content”: [  
{“type”: “image”, “url”: “https://…/candy.JPG”},  
{“type”: “text”, “text”: “What animal is on the candy?”}  
]  
},  
]

print(pipe(text=messages))

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [May 30, 2025, 11:37pm UTC](https://discuss.huggingface.co/t/valueerror-supplied-state-dict-for-layers-does-not-contain-bitsandbytes-and-possibly-other-quantized-stats-when-load-saved-quantized-model/110808/5 "2025-05-30T23:37:52Z")

</div>

> Is this issue fixed ?

Maybe not yet.