# OOM issue with large dataset streaming **URL:** \ **Category:** 🤗Datasets\ **Created:** [March 6, 2025, 3:56pm UTC](https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488 "2025-03-06T15:56:42Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![mayurdbs](https://avatars.discourse-cdn.com/v4/letter/m/e0b2c6/32.png) [@mayurdbs](https://discuss.huggingface.co/u/mayurdbs)\ **Post date:** [March 6, 2025, 3:56pm UTC](https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488/1 "2025-03-06T15:56:42Z")
For training, we load a multiple datasets (each with 1000+ 1GB+ each arrow files saved using dataset) with load\_dataset(…streaming=true) and then interleave them together with interleave\_dataset. Even with streaming dataset we are seeing OOM at the interleaving\_datasets step. Note no data has been read from datasets yet. Sharding the initial datasets per gpu before interleaving alleviated the memory pressure. Why should there be such memory issues with streaming dataset? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [March 6, 2025, 4:06pm UTC](https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488/2 "2025-03-06T16:06:22Z")
Is the data set file itself too large…? If it seems like a bug, let’s call lhonestq. > [@How to load a large hf dataset efficiently?](https://discuss.huggingface.co/t/how-to-load-a-large-hf-dataset-efficiently/69288): > > I am trying to load a dataset [axiong/pmc\_oa · Datasets at Hugging Face](https://huggingface.co/datasets/axiong/pmc_oa) The dataset size is around 22 gb and I have ram ~10 GB, the dataset object is stuck at extracting file point I also tried streaming mode but that’s giving another error. from datasets import load\_dataset dataset = load\_dataset("axiong/pmc\_oa", 'pmc\_oa', split='train', streaming=True) print(next(iter(dataset))) Any suggestion on how to deal with… ---
**Author:** ![mayurdbs](https://avatars.discourse-cdn.com/v4/letter/m/e0b2c6/32.png) [@mayurdbs](https://discuss.huggingface.co/u/mayurdbs)\ **Post date:** [March 6, 2025, 4:22pm UTC](https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488/3 "2025-03-06T16:22:48Z")
With each load\_dataset() we are loading ~1000 arrow files of ~ 1GB each. 15 such datasets are interleaved together. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [March 11, 2025, 3:27pm UTC](https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488/4 "2025-03-11T15:27:53Z")
if the iterable datasets have unknown feature types, `interleave_datasets` streams the first rows of the datasets to check the types and see whether the datasets can be interleaved without issues. But still this uses little memory (maximum one arrow record batch per dataset) so I can’t tell if this is the source of your OOM ---
**Author:** ![mayurdbs](https://avatars.discourse-cdn.com/v4/letter/m/e0b2c6/32.png) [@mayurdbs](https://discuss.huggingface.co/u/mayurdbs)\ **Post date:** [March 13, 2025, 6:23pm UTC](https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488/5 "2025-03-13T18:23:27Z")
is HF datasets recommended (streaming or otherwise) for such large scale dataset training? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [March 14, 2025, 5:24pm UTC](https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488/6 "2025-03-14T17:24:27Z")
Yep, especially streaming which is meant exactly for fast, large scale data loading 🙂 ---
**Author:** ![mayurdbs](https://avatars.discourse-cdn.com/v4/letter/m/e0b2c6/32.png) [@mayurdbs](https://discuss.huggingface.co/u/mayurdbs)\ **Post date:** [March 15, 2025, 3:48am UTC](https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488/7 "2025-03-15T03:48:03Z")
No, I meant given that iterable dataset is failing with large dataset size, has it been tried and tested to work with very large scale datasets?