# GPU memory usage is twice (2x) what I calculated based on number of parameters and floating point precision

**URL:** <https://discuss.huggingface.co/t/gpu-memory-usage-is-twice-2x-what-i-calculated-based-on-number-of-parameters-and-floating-point-precision/87300>\
**Category:** Intermediate\
**Created:** [May 18, 2024, 2:39pm UTC](https://discuss.huggingface.co/t/gpu-memory-usage-is-twice-2x-what-i-calculated-based-on-number-of-parameters-and-floating-point-precision/87300 "2024-05-18T14:39:59Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![clam004](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/clam004/32/26592_2.png) [@clam004](https://discuss.huggingface.co/u/clam004)\
**Post date:** [May 18, 2024, 2:39pm UTC](https://discuss.huggingface.co/t/gpu-memory-usage-is-twice-2x-what-i-calculated-based-on-number-of-parameters-and-floating-point-precision/87300/1 "2024-05-18T14:39:59Z")

</div>

When i do

```python
model = AutoModelForCausalLM.from_pretrained(
    "EleutherAI/gpt-neo-125m",
    low_cpu_mem_usage=True
)

```

I expect that since this is a fp32 model with 0.125 billion parameters, the amount of vRAM the model should occupy in GPU should be: 4 bytes per parameter x 0.125 billion parameters = 0.5 GB. Instead I see that 1000MiB are occupied when I do nvidia-smi, what am I missing?

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [May 18, 2024, 5:31pm UTC](https://discuss.huggingface.co/t/gpu-memory-usage-is-twice-2x-what-i-calculated-based-on-number-of-parameters-and-floating-point-precision/87300/2 "2024-05-18T17:31:00Z")

</div>

Can you share your full code? I’m seeing 529MB:

```python
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "EleutherAI/gpt-neo-125m",
    low_cpu_mem_usage=True,
)

model.to("cuda")

print(f"Used GPU memory: {torch.cuda.memory_allocated() / 1024 / 1024} MB")

```

> Used GPU memory: 529.86328125 MB

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [May 18, 2024, 5:32pm UTC](https://discuss.huggingface.co/t/gpu-memory-usage-is-twice-2x-what-i-calculated-based-on-number-of-parameters-and-floating-point-precision/87300/3 "2024-05-18T17:32:47Z")

</div>

Also do note that your GPU will reserve some space in it when the driver warms up. It’s better to use `torch.cuda.memory_allocated()` here.

E.g. just reserving a tiny tensor on the GPU will use 152MiB in nvidia-smi:

```python
import torch

t = torch.tensor([0.,1.]).cuda()

import time
time.sleep(10)

```

---

<div class="post-metadata">

**Author:** ![clam004](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/clam004/32/26592_2.png) [@clam004](https://discuss.huggingface.co/u/clam004)\
**Post date:** [May 18, 2024, 9:40pm UTC](https://discuss.huggingface.co/t/gpu-memory-usage-is-twice-2x-what-i-calculated-based-on-number-of-parameters-and-floating-point-precision/87300/4 "2024-05-18T21:40:45Z")

</div>

thanks @muellerzr, I also get `Used GPU memory: 529.86328125 MB` when i run

```python
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "EleutherAI/gpt-neo-125m",
    low_cpu_mem_usage=True,
)

model.to("cuda")

print(f"Used GPU memory: {torch.cuda.memory_allocated() / 1024 / 1024} MB")

```

But `nvidia-smi -l` reports `980MiB`, so i guess what youre saving is that my GPU is “reserving” twice the model’s parameters in memory. Whats this for?

More importantly, which one, `nvidia-smi -l ` or `torch.cuda.memory_allocated()`, is more indicative of when I am about to `torch.cuda.OutOfMemoryError`? Because at the end of the day, im just trying to extrapolate what hardware I need for a given model architecture, sequence\_length, batch size and optimizer.

thanks again!

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [May 18, 2024, 9:44pm UTC](https://discuss.huggingface.co/t/gpu-memory-usage-is-twice-2x-what-i-calculated-based-on-number-of-parameters-and-floating-point-precision/87300/5 "2024-05-18T21:44:06Z")

</div>

> [@clam004](#):
>
> so i guess what youre saving is that my GPU is “reserving” twice the model’s parameters in memory.

Nope that’s not what I’m saying at all. There’s certain overhead CUDA needs when doing things under the hood with all their drivers. It is far from 2x otherwise it’d be impossible to train some models 🙂 (And it’s all _usable_ memory that’s available, just might not be “in use”)

> [@clam004](#):
>
> torch.cuda.memory\_allocated()

This is more indicitive, however in general if you get cuda OOM that just means that again, you ran out of cuda memory. Looking at either or for hints won’t really per-se do much.

After you’ve gone through the initial parts (so like a step or two in) then you can eyeball the output on `nvidia-smi` (or GPU memory allocated % when looking at like W&B)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [May 19, 2024, 9:44am UTC](https://discuss.huggingface.co/t/gpu-memory-usage-is-twice-2x-what-i-calculated-based-on-number-of-parameters-and-floating-point-precision/87300/6 "2024-05-19T09:44:47Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.