# Errors running Inference Endpoint with quantized model

**URL:** <https://discuss.huggingface.co/t/errors-running-inference-endpoint-with-quantized-model/54444>\
**Category:** Inference Endpoints on the Hub\
**Created:** [September 11, 2023, 10:43am UTC](https://discuss.huggingface.co/t/errors-running-inference-endpoint-with-quantized-model/54444 "2023-09-11T10:43:00Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rfroli](https://avatars.discourse-cdn.com/v4/letter/r/b77776/32.png) [@rfroli](https://discuss.huggingface.co/u/rfroli)\
**Post date:** [September 11, 2023, 10:43am UTC](https://discuss.huggingface.co/t/errors-running-inference-endpoint-with-quantized-model/54444/1 "2023-09-11T10:43:00Z")

</div>

I’m encountering an issue while trying to load a quantized model into an Inference Endpoint. The model in question is “TheBloke/Llama-2-70B-chat-GPTQ”, and I’m using an AWS CPU Medium setup with PyTorch.

The error log suggests that the issue may be related to an incompatible version of `bitsandbytes`. Here’s the log for reference:

Error log :

2023/09/11 11:06:43 ~ Detected the presence of a `quantization_config` attribute in the model’s configuration but you don’t have the correct `bitsandbytes` version to support int8 serialization. Please install the latest version of `bitsandbytes` with `pip install --upgrade bitsandbytes`.  
2023/09/11 11:06:51 ~ entrypoint.sh: line 13: 7 Killed uvicorn webservice\_starlette:app --host 0.0.0.0 --port 5000

Does anyone have any suggestions on how to resolve this? The goal is to deploy lower-cost, CPU-based inference endpoints in production.

---

<div class="post-metadata">

**Author:** ![Hatman](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hatman/32/27670_2.png) [@Hatman](https://discuss.huggingface.co/u/Hatman)\
**Post date:** [September 12, 2023, 10:17pm UTC](https://discuss.huggingface.co/t/errors-running-inference-endpoint-with-quantized-model/54444/2 "2023-09-12T22:17:15Z")

</div>

Can you throw your .py into a container?

---

<div class="post-metadata">

**Author:** ![rfroli](https://avatars.discourse-cdn.com/v4/letter/r/b77776/32.png) [@rfroli](https://discuss.huggingface.co/u/rfroli)\
**Post date:** [September 14, 2023, 7:23am UTC](https://discuss.huggingface.co/t/errors-running-inference-endpoint-with-quantized-model/54444/3 "2023-09-14T07:23:24Z")

</div>

I’m just getting started in this area, I’m unsure how to retrieve the .py file, since the Inference Endpoint is generated automatically upon model import…