# Loading a large dataset occupies ~2GB on each GPU

**URL:** <https://discuss.huggingface.co/t/loading-a-large-dataset-occupies-2gb-on-each-gpu/83569>\
**Category:** 🤗Datasets\
**Created:** [April 24, 2024, 3:33pm UTC](https://discuss.huggingface.co/t/loading-a-large-dataset-occupies-2gb-on-each-gpu/83569 "2024-04-24T15:33:07Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jubueche](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jubueche/32/20460_2.png) [@jubueche](https://discuss.huggingface.co/u/jubueche)\
**Post date:** [April 24, 2024, 3:33pm UTC](https://discuss.huggingface.co/t/loading-a-large-dataset-occupies-2gb-on-each-gpu/83569/1 "2024-04-24T15:33:08Z")

</div>

Hi,

We are calling `dataset = load_dataset(dataset_name)` in a DDP setting, meaning that every GPU has its own process and executes this when it can.  
We see that after loading the dataset, every GPU has roughly 2GB allocated memory. Weirdly, this seems to appear only near the end of our training.  
During `load_dataset` we also see high GPU utilization when each GPUs process takes its turn. Why is that? I thought this function doesn’t use the GPU at all.

Any insights would be greatly appreciated.