# Best practices for a large dataset

**URL:** <https://discuss.huggingface.co/t/best-practices-for-a-large-dataset/137632>\
**Category:** 🤗Datasets\
**Created:** [January 27, 2025, 11:10pm UTC](https://discuss.huggingface.co/t/best-practices-for-a-large-dataset/137632 "2025-01-27T23:10:20Z")\
**Posts on this page:** 1\
**Showing post:** 8

<div class="post-metadata">

**Author:** ![nitishpandey04](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nitishpandey04/32/33204_2.png) [@nitishpandey04](https://discuss.huggingface.co/u/nitishpandey04)\
**Post date:** [May 6, 2025, 2:21pm UTC](https://discuss.huggingface.co/t/best-practices-for-a-large-dataset/137632/8 "2025-05-06T14:21:42Z")

</div>

Tell me, so you have implemented a pytorch `IterableDataset` that is calling 2 huggingface `IterableDataset`s for +ve and -ve class items right?

I think using individual iterable datasets for +ve and -ve classes can be a good idea but the wrapper you are using on top of those datasets doesn’t need to be iterable dataset itself. The parent can be a normal dataset. Have you tried this ?

Or maybe you can try to implement a single iterable dataset in pytorch which loads both +ve and -ve classes

I opened this issue - [Big text dataset loading for training](https://discuss.huggingface.co/t/big-text-dataset-loading-for-training/153576). Any insights you can share ?

---

_[View the full topic](https://discuss.huggingface.co/t/best-practices-for-a-large-dataset/137632)._