# Creating dataset slow **URL:** \ **Category:** 🤗Datasets\ **Created:** [December 18, 2024, 5:34am UTC](https://discuss.huggingface.co/t/creating-dataset-slow/132008 "2024-12-18T05:34:29Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![neilhavefun](https://avatars.discourse-cdn.com/v4/letter/n/71e660/32.png) [@neilhavefun](https://discuss.huggingface.co/u/neilhavefun)\ **Post date:** [December 18, 2024, 5:34am UTC](https://discuss.huggingface.co/t/creating-dataset-slow/132008/1 "2024-12-18T05:34:29Z")
I’m trying to create dataset by using .from\_generator(). At the beginning the first 1000 items is processed very fast. However, after 1000, the processing speed is extremely slow. I’m curious why this happen? I’m really appreciate if anyone can help me with this problem. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [December 18, 2024, 5:49am UTC](https://discuss.huggingface.co/t/creating-dataset-slow/132008/2 "2024-12-18T05:49:36Z")
Perhaps the data set has become too large, exceeding the capacity of the RAM, and you are using an SSD or HDD as a substitute for RAM? You might want to look for some know-how on creating large data sets. > [@How to create a new large Dataset on disk?](https://discuss.huggingface.co/t/how-to-create-a-new-large-dataset-on-disk/19616): > > I have my own library which processes corpora of documents much larger than what fits into memory. I would like to create a new HF Dataset by incrementally adding/streaming new examples to it, but because of the number of examples the dataset would not fit into memory. I am not sure if add\_item should get used for this as it does not seem to modify the dataset in place but rather returns a new dataset which looks like the dataset would actually be immutable and add\_item would create a new data… > [@Strategy for generating a large dataset](https://discuss.huggingface.co/t/strategy-for-generating-a-large-dataset/29063): > > Hi, I have to generate a dataset from 1,000+ large files by: making a random choice with replacement of a file per example (fast, this step takes a total of ~1 min for all examples). We need to keep a list of labels per file that describe some categories the file belongs to. sampling each of the chosen files at a random location (slow, ~a few days) and extracting a numerical vector per example Some constraints: the data is proprietary and the dataset cannot be uploaded online, it has to rem… ---
**Author:** ![neilhavefun](https://avatars.discourse-cdn.com/v4/letter/n/71e660/32.png) [@neilhavefun](https://discuss.huggingface.co/u/neilhavefun)\ **Post date:** [December 18, 2024, 6:50am UTC](https://discuss.huggingface.co/t/creating-dataset-slow/132008/3 "2024-12-18T06:50:04Z")
Thanks for the reply. Here are some details. I read data from hdf5 files. And trying to build a hf dataset for training purpose. The item of my dataset is contained with 3 images and labels. For instance, pseudo code like mygen(): files = readfiles(path) for hdf5\_item in files: with readfile(hdf5) : yield{{“img1”,img1},{“img2”,img2},{“img3”,img3}}. The resolutions of imgs are 640X1080X3. It contains larger than 40,000 items. I would appreciate it if you could provide some concrete examples or ideas for building a dataset like this. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [December 18, 2024, 7:15am UTC](https://discuss.huggingface.co/t/creating-dataset-slow/132008/4 "2024-12-18T07:15:44Z")
Anyway, I think that using from\_generator or from\_list will use up too much RAM. If you want to create a very large dataset, you can define a script to load the dataset, but if you can limit the dataset to images, you might be able to use the following method. In short, you can create a dataset just by uploading it to HF in separate directories. > **[Create an image dataset](https://huggingface.co/docs/datasets/v3.2.0/image_dataset)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[Create a dataset loading script](https://huggingface.co/docs/datasets/dataset_script)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ---
**Author:** ![neilhavefun](https://avatars.discourse-cdn.com/v4/letter/n/71e660/32.png) [@neilhavefun](https://discuss.huggingface.co/u/neilhavefun)\ **Post date:** [December 18, 2024, 7:30am UTC](https://discuss.huggingface.co/t/creating-dataset-slow/132008/5 "2024-12-18T07:30:19Z")
Thanks for the reply. Is from\_generator the most efficient way to create the large-scale dataset? I think the problem is about the RAM. I’m planning to create and save the dataset locally. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [December 18, 2024, 7:37am UTC](https://discuss.huggingface.co/t/creating-dataset-slow/132008/6 "2024-12-18T07:37:33Z")
I don’t have much experience with datasets libraries, so please consider this as a reference only.😅 **from\_** is simple and convenient, but I don’t think it’s suitable for creating large datasets. Unless you add more RAM to your PC… On the other hand, I think the method of creating on disk and then loading at the end, or writing a script dedicated to loading, is suitable for creating huge datasets. When reading the dataset, IterableDataset (streaming) should be available. This saves RAM by reducing the amount of data loaded at once. I don’t know how to use it when creating a dataset. > **[Stream](https://huggingface.co/docs/datasets/stream)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science.