# Mistral load\_in\_8bit slow inference

**URL:** <https://discuss.huggingface.co/t/mistral-load-in-8bit-slow-inference/88263>\
**Category:** 🤗Transformers\
**Created:** [May 24, 2024, 1:23pm UTC](https://discuss.huggingface.co/t/mistral-load-in-8bit-slow-inference/88263 "2024-05-24T13:23:26Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![swtb](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@swtb](https://discuss.huggingface.co/u/swtb)\
**Post date:** [May 24, 2024, 1:23pm UTC](https://discuss.huggingface.co/t/mistral-load-in-8bit-slow-inference/88263/1 "2024-05-24T13:23:26Z")

</div>

Hello,

I have run mistral against llama 3 for fp16, 4 bit, and 8bit quantisation.

in both cases 8 bit is dramatically slower. however, mistral is significantly slower for inference.

I use the “generate” method as opposed to pipelines as the docs say pipelines is not optimised for 8bit.

Can someone please explain why 8bit is still slow for inference, and why specifically on mistral?