# Low RAM Usage & high GPU usage, Datasets not helping

**URL:** <https://discuss.huggingface.co/t/low-ram-usage-high-gpu-usage-datasets-not-helping/29269>\
**Category:** 🤗Datasets\
**Created:** [January 10, 2023, 3:28am UTC](https://discuss.huggingface.co/t/low-ram-usage-high-gpu-usage-datasets-not-helping/29269 "2023-01-10T03:28:37Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sachinkalsi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sachinkalsi/32/698_2.png) [@sachinkalsi](https://discuss.huggingface.co/u/sachinkalsi)\
**Post date:** [January 10, 2023, 3:28am UTC](https://discuss.huggingface.co/t/low-ram-usage-high-gpu-usage-datasets-not-helping/29269/1 "2023-01-10T03:28:38Z")

</div>

Hello everyone, I’ve been following HF’s tutorials on “Fine-tuning a masked language model” [Fine-tuning a masked language model - Hugging Face Course](https://huggingface.co/course/en/chapter7/3).

I have got a sample training dataset of 20\_000 points, preprocessed according to the need. I’ve used datasets to read the data points & I am also using the data collate function to achieve dynamic masking for every batch.

I am using g4dn.2xlarge ([g4dn.2xlarge pricing and specs - Vantage](https://instances.vantage.sh/aws/ec2/g4dn.2xlarge) - 32GB RAM, 16GB GPU, 8 vCPUs) instance for fine-tuning `Roberta-base` MLM task with a brach size of 8 & each datapoint is of 512 sequence length.

With the above config, I observed GPU memory was very high, 95%+ and system RAM utilization was around 13-15%!

I did set follow the [Cache management — datasets 1.12.0 documentation](https://huggingface.co/docs/datasets/v1.12.0/cache.html) & set `IN_MEMORY_MAX_SIZE` to ~25GB, but no luck

```auto
import datasets
datasets.config.IN_MEMORY_MAX_SIZE = 24_696_061_952
train_dataset = load_dataset('pandas', data_files={'train': 'path to pickle file'}, keep_in_memory=True)

```

But RAM usage remained as it is. How can I make full utilization of RAM & GPU memory?

I’ve to take 20\_000 as the sample for this experiment but I’ve ~1 Million data points, which I will be using in the future for full-fledged training, post-resolution to this problem

Thanks

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [January 13, 2023, 4:31pm UTC](https://discuss.huggingface.co/t/low-ram-usage-high-gpu-usage-datasets-not-helping/29269/2 "2023-01-13T16:31:23Z")

</div>

How big is your dataset in bytes ? By adding `keep_in_memory=True` you load it completely in memory.

20,000 is rather small: 20,000 points \* 512 tokens \* 4 bytes ~= 41MB

---

<div class="post-metadata">

**Author:** ![sachinkalsi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sachinkalsi/32/698_2.png) [@sachinkalsi](https://discuss.huggingface.co/u/sachinkalsi)\
**Post date:** [January 13, 2023, 4:46pm UTC](https://discuss.huggingface.co/t/low-ram-usage-high-gpu-usage-datasets-not-helping/29269/3 "2023-01-13T16:46:30Z")

</div>

Thanks for the response.

yes. You are right. Its 41MB. I have a total dataset of ~7GB.

With 16GB GPU, the maximum I can do is 8 batch sizes. If I want to use 16 batch size, then I need 32+GB GPU, but the instance would also have heavy main memory (RAM). i.e., around 64GB or 128GB, etc. But I will be using only a fraction of it (RAM) but full utilization of GPU.

This is the first time I am training MLM, not sure whether this is usually the case.

thanks again

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [January 13, 2023, 4:52pm UTC](https://discuss.huggingface.co/t/low-ram-usage-high-gpu-usage-datasets-not-helping/29269/4 "2023-01-13T16:52:57Z")

</div>

You don’t need to fill up your RAM to get the best data loading throughput while training. If your dataset fits in RAM you’re already all set ^^