# How to speed up CodeLlama inference?

**URL:** <https://discuss.huggingface.co/t/how-to-speed-up-codellama-inference/96913>\
**Category:** Beginners\
**Created:** [July 12, 2024, 2:15am UTC](https://discuss.huggingface.co/t/how-to-speed-up-codellama-inference/96913 "2024-07-12T02:15:34Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![QGeorgeL](https://avatars.discourse-cdn.com/v4/letter/q/dbc845/32.png) [@QGeorgeL](https://discuss.huggingface.co/u/QGeorgeL)\
**Post date:** [July 12, 2024, 2:15am UTC](https://discuss.huggingface.co/t/how-to-speed-up-codellama-inference/96913/1 "2024-07-12T02:15:34Z")

</div>

I’m running CodeLlama 7B on 8\*A100 PCIE 40G.

```python
q_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_use_double_quant=True,
   bnb_4bit_compute_dtype=torch.bfloat16
) 
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map = "auto",
    quantization_config=q_config,
    max_memory = {0:"39GB",1:"39GB",2:"39GB",3:"39GB",4:"39GB",5:"39GB",6:"39GB",7:"39GB"}
)

```

With `max_new_tokens=4096` Each inference takes 20 seconds.  
I’m wondering if there are other ways to speed up inference besides lowering `max_new_tokens`.  
Thanks