# How to run large LLMs like Llama 3.1 70B or Mixtral 8x22B with limited GPU VRAM?

**URL:** <https://discuss.huggingface.co/t/how-to-run-large-llms-like-llama-3-1-70b-or-mixtral-8x22b-with-limited-gpu-vram/108539>\
**Category:** Beginners\
**Created:** [September 25, 2024, 3:49am UTC](https://discuss.huggingface.co/t/how-to-run-large-llms-like-llama-3-1-70b-or-mixtral-8x22b-with-limited-gpu-vram/108539 "2024-09-25T03:49:35Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![srivassid](https://avatars.discourse-cdn.com/v4/letter/s/d2c977/32.png) [@srivassid](https://discuss.huggingface.co/u/srivassid)\
**Post date:** [September 25, 2024, 3:49am UTC](https://discuss.huggingface.co/t/how-to-run-large-llms-like-llama-3-1-70b-or-mixtral-8x22b-with-limited-gpu-vram/108539/1 "2024-09-25T03:49:35Z")

</div>

I am trying to run large LLMs like LLama 3.1 70B and Mixtral 8x22B locally on my system. I have 3x3090 and 4080 Super 16GB.

I read on reddit that people were able to run those models on a single 3090 with 4 bit quantization. I am trying to run the same models using 4 bit quantizations but have been unsuccessful so far.

I tried to run [this](https://huggingface.co/unsloth/Meta-Llama-3.1-70B-Instruct-bnb-4bit) model for LLama and [this](https://huggingface.co/mistral-community/Mixtral-8x22B-Instruct-v0.1-4bit) model for mixtral. I also tried to run the base model by llama by passing 4 bit quantization as a parameter but no dice.

I was successfully able to run LLama 3.1 7B instruct and Mixtral 7B instruct locally (without 4 bit quantization).

Anyone has any idea how can i run those models?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [September 25, 2024, 4:49am UTC](https://discuss.huggingface.co/t/how-to-run-large-llms-like-llama-3-1-70b-or-mixtral-8x22b-with-limited-gpu-vram/108539/2 "2024-09-25T04:49:11Z")

</div>

You mean you dare to use a quantized model expanded into a float? In any case, those models are not gated, and even a single GPU should be enough. My GPU is much crappier than yours even when compared to one GPU to another. Still, the quantized one actually worked.

```auto
pip install bitsandbytes
pip install accelerate

```

```py
from transformers import BitsAndBytesConfig
nf4_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                               bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)

model_nf4 = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=nf4_config, device_map="auto", torch_dtype=torch.bfloat16).eval()

model_bf16 = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype=torch.bfloat16).eval()

```

---

<div class="post-metadata">

**Author:** ![srivassid](https://avatars.discourse-cdn.com/v4/letter/s/d2c977/32.png) [@srivassid](https://discuss.huggingface.co/u/srivassid)\
**Post date:** [September 26, 2024, 3:52am UTC](https://discuss.huggingface.co/t/how-to-run-large-llms-like-llama-3-1-70b-or-mixtral-8x22b-with-limited-gpu-vram/108539/3 "2024-09-26T03:52:51Z")

</div>

Ok i will give it a try…thanks