# Streamlit + Llama 3, takes too much gpu memory?

**URL:** <https://discuss.huggingface.co/t/streamlit-llama-3-takes-too-much-gpu-memory/97193>\
**Category:** Models\
**Created:** [July 13, 2024, 10:36pm UTC](https://discuss.huggingface.co/t/streamlit-llama-3-takes-too-much-gpu-memory/97193 "2024-07-13T22:36:50Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Senthy](https://avatars.discourse-cdn.com/v4/letter/s/f07891/32.png) [@Senthy](https://discuss.huggingface.co/u/Senthy)\
**Post date:** [July 13, 2024, 10:36pm UTC](https://discuss.huggingface.co/t/streamlit-llama-3-takes-too-much-gpu-memory/97193/1 "2024-07-13T22:36:50Z")

</div>

Thanks for taking the time out of your day to read this,

I had integrated meta-llama/Meta-Llama-3-8B-Instruct, on to my pc, which went perfect. I had decided I had wanted to integrate a streamlit ui, to make it easier to access this. However I came upon a plethora of issues, mainly regarding the quantization of such. It stated I didn’t have enough ram, I was wondering if there are any settings I can alter within my code to make this possible, or if i should just use a different model. I do understand that Llama 3 is a huge model, but does streamlit push the llama model too much where it cant even make a ui?

Here are my quantization settings using bnb config: bnb\_config = BitsAndBytesConfig(  
load\_in\_4bit=True,  
bnb\_4bit\_use\_double\_quant=True,  
bnb\_4bit\_quant\_type=‘nf4’,  
bnb\_4bit\_compute\_dtype=torch.bfloat16  
)  
and here is my text generator class:  
text\_generator = pipeline(  
task=“text-generation”,  
model=model,  
tokenizer=tokenizer,  
max\_new\_tokens=64, \*note this was originally 128, but i decreased this  
)  
I had tried implementing loading in 8 bit fp 32, but it seems that the llama model doesn’t support that.

As I end this post, I have a 2060 super with 8 dedicated gpu memory, any tips would be helpful.  
Thankful for your time!