# Enabling load\_in\_8bit makes inference much slower

**URL:** <https://discuss.huggingface.co/t/enabling-load-in-8bit-makes-inference-much-slower/38596>\
**Category:** 🤗Transformers\
**Created:** [May 3, 2023, 9:36pm UTC](https://discuss.huggingface.co/t/enabling-load-in-8bit-makes-inference-much-slower/38596 "2023-05-03T21:36:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![feizheng](https://avatars.discourse-cdn.com/v4/letter/f/ed655f/32.png) [@feizheng](https://discuss.huggingface.co/u/feizheng)\
**Post date:** [May 3, 2023, 9:36pm UTC](https://discuss.huggingface.co/t/enabling-load-in-8bit-makes-inference-much-slower/38596/1 "2023-05-03T21:36:35Z")

</div>

I loaded the 7b llama on an A100 this way:

```auto
quantization_config = transformers.BitsAndBytesConfig(load_in_8bit=False, llm_int8_threshold=0.0)
model = LlamaForCausalLM.from_pretrained(
   "decapoda-research/llama-7b-hf",
    device_map="auto",
    torch_dtype=torch.float16,
    quantization_config=quantization_config,
)

```

When `load_in_8bit` is `False`, it generates 16.7 tokens/sec whereas `load_in_8bit=True` generates only 6.7 tokens/sec. It seems I probably set up configurations incorrectly for `load_in_8bit=True`.

My transformers version is `4.29.0.dev0`. Did I miss anything? Thanks.

---

<div class="post-metadata">

**Author:** ![abhishek-wfx](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@abhishek-wfx](https://discuss.huggingface.co/u/abhishek-wfx)\
**Post date:** [August 30, 2023, 1:09pm UTC](https://discuss.huggingface.co/t/enabling-load-in-8bit-makes-inference-much-slower/38596/2 "2023-08-30T13:09:58Z")

</div>

I am facing similar issue, did you find a solution?

---

<div class="post-metadata">

**Author:** ![chaochaoli](https://avatars.discourse-cdn.com/v4/letter/c/9fc348/32.png) [@chaochaoli](https://discuss.huggingface.co/u/chaochaoli)\
**Post date:** [September 9, 2023, 7:48am UTC](https://discuss.huggingface.co/t/enabling-load-in-8bit-makes-inference-much-slower/38596/3 "2023-09-09T07:48:03Z")

</div>

me too，and training is slower too

---

<div class="post-metadata">

**Author:** ![AbhijitHerekar](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AbhijitHerekar](https://discuss.huggingface.co/u/AbhijitHerekar)\
**Post date:** [February 13, 2024, 7:05am UTC](https://discuss.huggingface.co/t/enabling-load-in-8bit-makes-inference-much-slower/38596/4 "2024-02-13T07:05:40Z")

</div>

same here, not sure if anyone found the root cause.