# Memory overhead/usage calculation

**URL:** <https://discuss.huggingface.co/t/memory-overhead-usage-calculation/159712>\
**Category:** Intermediate\
**Created:** [June 17, 2025, 7:35pm UTC](https://discuss.huggingface.co/t/memory-overhead-usage-calculation/159712 "2025-06-17T19:35:10Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![tiago-machado](https://avatars.discourse-cdn.com/v4/letter/t/58956e/32.png) [@tiago-machado](https://discuss.huggingface.co/u/tiago-machado)\
**Post date:** [June 17, 2025, 7:35pm UTC](https://discuss.huggingface.co/t/memory-overhead-usage-calculation/159712/1 "2025-06-17T19:35:10Z")

</div>

Hello,

Is there a way to find out how much GPU memory an inference is using?  
For example, giving that I have a text of 256 tokens, when I input it to the LLM, asking it to generate a response with 256 tokens (maximum), how can I know how much GPU memory was used for the task?

---

<div class="post-metadata">

**Author:** ![Mdrnfox](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mdrnfox/32/47695_2.png) [@Mdrnfox](https://discuss.huggingface.co/u/Mdrnfox)\
**Post date:** [June 17, 2025, 10:20pm UTC](https://discuss.huggingface.co/t/memory-overhead-usage-calculation/159712/2 "2025-06-17T22:20:49Z")

</div>

> [@tiago-machado](#):
>
> Is there a way to find out how much GPU memory an inference is using?  
> For example, giving that I have a text of 256 tokens, when I input it to the LLM, asking it to generate a response with 256 tokens (maximum), how can I know how much GPU memory was used for the task?

torch.cuda.reset\_peak\_memory\_stats()

start\_mem = torch.cuda.memory\_allocated()

# Run inference

output = model.generate(input\_ids, max\_new\_tokens=256)

# After inference

end\_mem = torch.cuda.memory\_allocated()

peak\_mem = torch.cuda.max\_memory\_allocated()

---

<div class="post-metadata">

**Author:** ![tiago-machado](https://avatars.discourse-cdn.com/v4/letter/t/58956e/32.png) [@tiago-machado](https://discuss.huggingface.co/u/tiago-machado)\
**Post date:** [June 20, 2025, 4:50am UTC](https://discuss.huggingface.co/t/memory-overhead-usage-calculation/159712/3 "2025-06-20T04:50:50Z")

</div>

Hello, @Mdrnfox ! Thanks for your reply. Do you know the unit that torch is using, is it bits or bytes?

---

<div class="post-metadata">

**Author:** ![Mdrnfox](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mdrnfox/32/47695_2.png) [@Mdrnfox](https://discuss.huggingface.co/u/Mdrnfox)\
**Post date:** [June 20, 2025, 10:02am UTC](https://discuss.huggingface.co/t/memory-overhead-usage-calculation/159712/4 "2025-06-20T10:02:59Z")

</div>

I believe it is bytes. Element\_size() is in bytes