# GPU usage increasing every loop when running inference

**URL:** <https://discuss.huggingface.co/t/gpu-usage-increasing-every-loop-when-running-inference/51113>\
**Category:** Beginners\
**Created:** [August 17, 2023, 2:40pm UTC](https://discuss.huggingface.co/t/gpu-usage-increasing-every-loop-when-running-inference/51113 "2023-08-17T14:40:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zawedcvg](https://avatars.discourse-cdn.com/v4/letter/z/278dde/32.png) [@Zawedcvg](https://discuss.huggingface.co/u/Zawedcvg)\
**Post date:** [August 17, 2023, 2:40pm UTC](https://discuss.huggingface.co/t/gpu-usage-increasing-every-loop-when-running-inference/51113/1 "2023-08-17T14:40:15Z")

</div>

I have been trying to run inference on llama 2-13b with the following code on colab.

with torch.no\_grad():  
for context, multi, single, second in zip(all\_items, multi\_hop\_items, single\_hop\_items, second\_hop\_items):

```
inputs_multi = tokenizer(prompt_multi, return_tensors="pt").to("cuda")
generated_ids_multi = model.generate(**inputs_multi, max_length=4096)

outputs_multi = tokenizer.batch_decode(generated_ids_multi, skip_special_tokens=True)

answer_multi = outputs_multi[0]

```

There is increase in GPU ram in pretty much everyloop. And once say I stop the cell running in the middle, the GPU used stays at that level, leading me to go out of memory very quickly. i have tried using pipeline on dataset, based on the huggingface website and I seem to be having a very similar issue for that as well.

---

<div class="post-metadata">

**Author:** ![aschroeder91](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aschroeder91/32/22625_2.png) [@aschroeder91](https://discuss.huggingface.co/u/aschroeder91)\
**Post date:** [March 2, 2024, 8:12pm UTC](https://discuss.huggingface.co/t/gpu-usage-increasing-every-loop-when-running-inference/51113/2 "2024-03-02T20:12:54Z")

</div>

I am having a similar issue and do not have a solution.  
I use with torch.inference\_mode() and torch.no\_grad():  
using huggingface transformer library I use start a thread for model.generate for a transformer LLM  
After inference is complete my GPU monitor shows that utilization remains \>98% for the next 4-10 seconds and then returns to 0%.  
Also, consecutive inferences severely throttles generation speed to approximately 10x the generation time

---

<div class="post-metadata">

**Author:** ![bxrjmfh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bxrjmfh/32/45714_2.png) [@bxrjmfh](https://discuss.huggingface.co/u/bxrjmfh)\
**Post date:** [May 13, 2024, 3:59am UTC](https://discuss.huggingface.co/t/gpu-usage-increasing-every-loop-when-running-inference/51113/3 "2024-05-13T03:59:16Z")

</div>

same problem