# Big text dataset loading for training

**URL:** <https://discuss.huggingface.co/t/big-text-dataset-loading-for-training/153576>\
**Category:** 🤗Datasets\
**Created:** [May 5, 2025, 1:25pm UTC](https://discuss.huggingface.co/t/big-text-dataset-loading-for-training/153576 "2025-05-05T13:25:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nitishpandey04](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nitishpandey04/32/33204_2.png) [@nitishpandey04](https://discuss.huggingface.co/u/nitishpandey04)\
**Post date:** [May 5, 2025, 1:25pm UTC](https://discuss.huggingface.co/t/big-text-dataset-loading-for-training/153576/1 "2025-05-05T13:25:27Z")

</div>

I want to load a big dataset which is in shards of .txt files. Total dataset size is around 250GB.

If I just load it using `load_dataset("text", data_files=[])`, then it will load the complete dataset into the memory or it will use Arrow to do memory mapping and reduce the memory footprint of the dataset ?

And If I want to use the dataset for training, is it advisable to directly use Dataset or IterableDataset ?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [May 5, 2025, 9:48pm UTC](https://discuss.huggingface.co/t/big-text-dataset-loading-for-training/153576/2 "2025-05-05T21:48:25Z")

</div>

If you have excess VRAM or RAM, it seems best to use `Dataset`, but if not, it seems best to use `IterableDataset` with `num_workers=` increased.

> [@Best practices for a large dataset](https://discuss.huggingface.co/t/best-practices-for-a-large-dataset/137632):
>
> Hi, I have a ~1TB large dataset stored at the HF hub. I can download this on my disk and read it successfully. Nevertheless, it’s large enough that I can’t fit it in my RAM. What is the best practice for training a model on such a dataset? I tried loading the dataset with load\_dataset(..., streaming=True) and then having two buffers: one that is being loaded by training process onto GPU and one that, in a separate thread, streams from the dataset and fills up the other buffer. Then, when the i…

> <https://github.com/huggingface/datasets/issues/4235>
>
> \### System Info
> 
> \`\`\`shell
> I am using transformer trainer while meeting the issue….
> The trainer requests torch.utils.data.Dataset as input, which loads the whole dataset into the memory at once. Therefore, when the dataset is too large to load, there's nothing I can do except using IterDataset, which loads samples of data seperately, and results in low efficiency. 
> I wonder if there are any tricks like Sharding in huggingface trainer.
> Looking forward to your reply.
> \`\`\`
> 
> 
> \### Who can help?
> 
> Trainer: @sgugger
> 
> \### Information
> 
> \- \[\] The official example scripts
> \- \[\] My own modified scripts
> 
> \### Tasks
> 
> \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> None
> 
> \### Expected behavior
> 
> \`\`\`shell
> I wonder if there are any tricks like fairseq Sharding very large datasets https://fairseq.readthedocs.io/en/latest/getting\_started.html.
> Thanks a lot!
> \`\`\`

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [May 7, 2025, 12:41pm UTC](https://discuss.huggingface.co/t/big-text-dataset-loading-for-training/153576/3 "2025-05-07T12:41:05Z")

</div>

Hi ! `load_dataset()` converts the .txt files to arrow and memory maps the arrow file on disk, so yes it doesn’t fill your RAM.

> And If I want to use the dataset for training, is it advisable to directly use Dataset or IterableDataset ?

Depends on your dataset size, Dataset is perfectly fine for \<1TB. For bigger datasets IterableDataset provides faster initialization (no conversion needed) and saves disk space