# Hardware Requirement GPU

**URL:** <https://discuss.huggingface.co/t/hardware-requirement-gpu/137480>\
**Category:** Beginners\
**Created:** [January 27, 2025, 2:58am UTC](https://discuss.huggingface.co/t/hardware-requirement-gpu/137480 "2025-01-27T02:58:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![muzanxdem](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muzanxdem/32/36242_2.png) [@muzanxdem](https://discuss.huggingface.co/u/muzanxdem)\
**Post date:** [January 27, 2025, 2:58am UTC](https://discuss.huggingface.co/t/hardware-requirement-gpu/137480/1 "2025-01-27T02:58:43Z")

</div>

Hi, just need you guys opinion,

I have a chatbot system that run using below model:

- llama 3.2 90B
- llama 3.2 3B
- Whisper (large)
- nomic-embed-text (Embedding)

What is the best requirement cluster GPU needed to run above model?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [January 27, 2025, 3:48am UTC](https://discuss.huggingface.co/t/hardware-requirement-gpu/137480/2 "2025-01-27T03:48:19Z")

</div>

Apart from Llama 90B, the others are small (I don’t think they add up to 10GB), so we’ll look at the operating conditions for Llama 90B. At least 64GB of VRAM is required in 4-bit quantization. If you want to use fp16, you’ll need 256GB. In addition to loading the model, a little extra VRAM is required during inference, so it’s best to secure a little more VRAM than the model size.

There is a big difference in cost between 64GB and 256GB, so it’s better to run it in a quantized state (GGUF or NF4) somehow. Well, if you’re using Ollama or Llamacpp as a server, you just need to use the Q4\_K\_M format GGUF.

---

<div class="post-metadata">

**Author:** ![muzanxdem](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muzanxdem/32/36242_2.png) [@muzanxdem](https://discuss.huggingface.co/u/muzanxdem)\
**Post date:** [January 27, 2025, 3:55am UTC](https://discuss.huggingface.co/t/hardware-requirement-gpu/137480/3 "2025-01-27T03:55:03Z")

</div>

Hi @John6666  
Thank you for your response!

I am currently using Ollama to run Llama models. Would a cluster of four NVIDIA RTX A6000 GPUs be sufficient to handle the model?

Thanks!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [January 27, 2025, 4:05am UTC](https://discuss.huggingface.co/t/hardware-requirement-gpu/137480/4 "2025-01-27T04:05:39Z")

</div>

It has 192GB of VRAM!😆  
Ollama’s default is Q4\_K\_M (if you don’t specify otherwise, it uses this, which is unlikely to cause problems), so I think it will be more than enough. The amount of consumption is a little over 64GB for the model, plus a little for inference. If there are a lot of people using it at the same time, or if you want to process very long sentences, it will use more VRAM, but it will still be unlikely to cause problems.  
By the way, Ollama is also fast enough, but Llamacpp seems to be even faster. If you’re having trouble with speed, you might want to try changing it.

> **[@mitkox on Hugging Face: "llama.cpp is 26.8% faster than ollama. 
I have...](https://huggingface.co/posts/mitkox/389008233017077)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.