# 4bit quantization on inference end point

**URL:** <https://discuss.huggingface.co/t/4bit-quantization-on-inference-end-point/69366>\
**Category:** Inference Endpoints on the Hub\
**Created:** [January 16, 2024, 1:06pm UTC](https://discuss.huggingface.co/t/4bit-quantization-on-inference-end-point/69366 "2024-01-16T13:06:09Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![dbshift](https://avatars.discourse-cdn.com/v4/letter/d/ba8739/32.png) [@dbshift](https://discuss.huggingface.co/u/dbshift)\
**Post date:** [January 16, 2024, 1:06pm UTC](https://discuss.huggingface.co/t/4bit-quantization-on-inference-end-point/69366/1 "2024-01-16T13:06:09Z")

</div>

I need to know is there a way to quantize a model in 4bit and run it in inference end point. currently it is taking 8bit quantization as default. one more thing is can I do my custom quantization in handler.py file?