# Load quantized model in memory

**URL:** <https://discuss.huggingface.co/t/load-quantized-model-in-memory/58383>\
**Category:** Beginners\
**Created:** [October 13, 2023, 1:54am UTC](https://discuss.huggingface.co/t/load-quantized-model-in-memory/58383 "2023-10-13T01:54:38Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![fzanartu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fzanartu/32/27172_2.png) [@fzanartu](https://discuss.huggingface.co/u/fzanartu)\
**Post date:** [October 13, 2023, 1:54am UTC](https://discuss.huggingface.co/t/load-quantized-model-in-memory/58383/1 "2023-10-13T01:54:38Z")

</div>

Hey guys,

I’m looking for some guidance on a particular issue.  
I’m not using HF’s trainer, but my own pytorch implementation, the workflow is like:

```python
best_score = 0
for epoch in epochs:
    # Training
    # ...
    if metric > best_score:
        best_model = deepcopy(model.state_dict)

```

And then, to evaluate the best model:

`model.load_state_dict(best_model)`

This approach works perfectly when `model` is defined as `AutoModelForSequenceClassification.from_pretrained(modelcp, labels).` However, when I define `model` as `AutoModelForSequenceClassification.from_pretrained(modelcp, labels, load_in_8bit=True),` I encounter the following error:

RuntimeError: Loading a quantized checkpoint into non-quantized Linear8bitLt is not supported. Please call module.cuda() before module.load\_state\_dict()

Anyone could have any insights on how to resolve this issue? Cheers!

---

<div class="post-metadata">

**Author:** ![AngusHuang](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/angushuang/32/21975_2.png) [@AngusHuang](https://discuss.huggingface.co/u/AngusHuang)\
**Post date:** [December 8, 2023, 2:09am UTC](https://discuss.huggingface.co/t/load-quantized-model-in-memory/58383/2 "2023-12-08T02:09:09Z")

</div>

I have the same issue, anyone could help?