# AutoModelForCausalLM error with accelerate and bitsandbytes

**URL:** <https://discuss.huggingface.co/t/automodelforcausallm-error-with-accelerate-and-bitsandbytes/78966>\
**Category:** 🤗Accelerate\
**Created:** [March 27, 2024, 4:15pm UTC](https://discuss.huggingface.co/t/automodelforcausallm-error-with-accelerate-and-bitsandbytes/78966 "2024-03-27T16:15:02Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![altrastorique](https://avatars.discourse-cdn.com/v4/letter/a/f9ae1b/32.png) [@altrastorique](https://discuss.huggingface.co/u/altrastorique)\
**Post date:** [March 27, 2024, 4:15pm UTC](https://discuss.huggingface.co/t/automodelforcausallm-error-with-accelerate-and-bitsandbytes/78966/1 "2024-03-27T16:15:02Z")

</div>

I Use google colab but connected localy with my computer using jupyter. I have Windows 10, RTX 3070 and no cuda or cudnn because I didn’t succed to make it works ☹

### Reproduction

```auto
!pip install transformers trl accelerate torch bitsandbytes peft datasets -qU
!pip install flash-attn --no-build-isolation

from datasets import load_dataset

instruct_tune_dataset = load_dataset("mosaicml/instruct-v3")

...

model_id = "mistralai/Mixtral-8x7B-v0.1"

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

nf4_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map='auto',
quantization_config=nf4_config,
use_cache=False,
attn_implementation="flash_attention_2"

)

```

Error message:  
ImportError: Using `bitsandbytes` 8-bit quantization requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes: `pip install -i https://pypi.org/simple/ bitsandbytes`

### Expected behavior

This code works when using the google colab free T4 but don’t when I run the code localy. Even if I have installed bitsandbytes and accelerate.

Please help me 🙂  
Thank you very much!

---

<div class="post-metadata">

**Author:** ![marcsun13](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcsun13/32/19059_2.png) [@marcsun13](https://discuss.huggingface.co/u/marcsun13)\
**Post date:** [April 15, 2024, 1:31pm UTC](https://discuss.huggingface.co/t/automodelforcausallm-error-with-accelerate-and-bitsandbytes/78966/2 "2024-04-15T13:31:17Z")

</div>

Hi @altrastorique, bnb quantization requires you to have cuda or cudnn to work. The error that you are facing is probably due to that.