# Inference time in TGI quantization

**URL:** <https://discuss.huggingface.co/t/inference-time-in-tgi-quantization/87619>\
**Category:** Intermediate\
**Created:** [May 21, 2024, 9:21am UTC](https://discuss.huggingface.co/t/inference-time-in-tgi-quantization/87619 "2024-05-21T09:21:04Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![xavier7778](https://avatars.discourse-cdn.com/v4/letter/x/5e9695/32.png) [@xavier7778](https://discuss.huggingface.co/u/xavier7778)\
**Post date:** [May 21, 2024, 9:21am UTC](https://discuss.huggingface.co/t/inference-time-in-tgi-quantization/87619/1 "2024-05-21T09:21:04Z")

</div>

I have been inferencing a 7B llm model using TGI on runpod. till now i have been directly loading the model without quantisation.  
But recently when i tried quantisation of model on 4bit and 8bit using TGI on same gpu configurations. After quantisation the model was taking more inference time in both 8-bit and 4-bit.  
I was using bitsandbytes for quantisation.

Is this correct or am i missing something? shouldn’t the model inference time also reduce?