# Can i create endpoint using quantized model?

**URL:** <https://discuss.huggingface.co/t/can-i-create-endpoint-using-quantized-model/56416>\
**Category:** Inference Endpoints on the Hub\
**Created:** [September 26, 2023, 10:49am UTC](https://discuss.huggingface.co/t/can-i-create-endpoint-using-quantized-model/56416 "2023-09-26T10:49:16Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shamito](https://avatars.discourse-cdn.com/v4/letter/s/4bbf92/32.png) [@Shamito](https://discuss.huggingface.co/u/Shamito)\
**Post date:** [September 26, 2023, 10:49am UTC](https://discuss.huggingface.co/t/can-i-create-endpoint-using-quantized-model/56416/1 "2023-09-26T10:49:16Z")

</div>

hello everyone, I’m trying to create endpoint using quantied model ''TheBloke/llama-2-7B-chat-GPTQ" but facing errors.

So I’m wondering if i could create endpoint using it!

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 26, 2023, 11:53am UTC](https://discuss.huggingface.co/t/can-i-create-endpoint-using-quantized-model/56416/2 "2023-09-26T11:53:16Z")

</div>

What errors are you facing? Are you selection “gptq” in the advanced section?

---

<div class="post-metadata">

**Author:** ![chrislevy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrislevy/32/20797_2.png) [@chrislevy](https://discuss.huggingface.co/u/chrislevy)\
**Post date:** [October 31, 2023, 4:12pm UTC](https://discuss.huggingface.co/t/can-i-create-endpoint-using-quantized-model/56416/3 "2023-10-31T16:12:47Z")

</div>

Our organization is also testing out HF inference endpoints.

It’s not totally clear from the documentation how to run quantized models.  
For example if I select the model `mistral-7b-instruct-v0-1` and select GPTQ in the container configuration, will it work? I just see errors in the logs and failings to update.

And what if I choose a quantized model to begin with such as `TheBloke/Mistral-7B-Instruct-v0.1-GPTQ` or `TheBloke/Mistral-7B-OpenOrca-AWQ`, what do I select in the containter config for quantization? I assume None.

In either case I have not been able to successfully get an endpoint to work yet with a quantized model.

Like it’s not clear whether “GPTQ in the container configuration” is saying quantize this model or if its saying I chose a quantized model so make sure to check this box.

---

<div class="post-metadata">

**Author:** ![iarbel](https://avatars.discourse-cdn.com/v4/letter/i/3da27b/32.png) [@iarbel](https://discuss.huggingface.co/u/iarbel)\
**Post date:** [January 16, 2024, 5:59pm UTC](https://discuss.huggingface.co/t/can-i-create-endpoint-using-quantized-model/56416/4 "2024-01-16T17:59:18Z")

</div>

Were you able to get an answer for this? 🤔