# Extremely slow data loading of imagefolder

**URL:** <https://discuss.huggingface.co/t/extremely-slow-data-loading-of-imagefolder/60131>\
**Category:** 🤗Datasets\
**Created:** [October 27, 2023, 7:44am UTC](https://discuss.huggingface.co/t/extremely-slow-data-loading-of-imagefolder/60131 "2023-10-27T07:44:04Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![sienna223](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@sienna223](https://discuss.huggingface.co/u/sienna223)\
**Post date:** [October 27, 2023, 7:44am UTC](https://discuss.huggingface.co/t/extremely-slow-data-loading-of-imagefolder/60131/1 "2023-10-27T07:44:04Z")

</div>

Hi, I’m new to the Huggingface’s Datasets and I’m trying to train controlnet for stablediffusion on a custom dataset with approximately 300k images, the size of these images is (768, 768).

Now, I stuck in following lines of code:

```python
dataset = load_dataset("imagefolder", data_dir="path/to/the/dataset")
print(dataset['train'][0])

```

Then, I have few questions.

1. Does imagefolder load images (load and decode) in memory at setup, if it is, can I disable it?
2. Are there any implicit process Datasets do when I first call load\_dataset, so it takes that long time?
3. What’s the best practice to load a relatively large dataset? I see someone mention that saving dataset as Arrow and then load it, but I don’t know how to do it specifically. There is a urgent need for a detailed tutorial in official docs for this.

---

<div class="post-metadata">

**Author:** ![kopyl](https://avatars.discourse-cdn.com/v4/letter/k/3bc359/32.png) [@kopyl](https://discuss.huggingface.co/u/kopyl)\
**Post date:** [December 10, 2023, 12:32am UTC](https://discuss.huggingface.co/t/extremely-slow-data-loading-of-imagefolder/60131/2 "2023-12-10T00:32:12Z")

</div>

Agree. 3 hours in, still not even a log line -\_-

I have 3 million images in my directory…

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/e/1/e197ce9d69389e27b84227ae713fc7968ec7d3de.png)

---

<div class="post-metadata">

**Author:** ![panigrah](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@panigrah](https://discuss.huggingface.co/u/panigrah)\
**Post date:** [December 10, 2023, 1:01am UTC](https://discuss.huggingface.co/t/extremely-slow-data-loading-of-imagefolder/60131/3 "2023-12-10T01:01:55Z")

</div>

Does passing `streaming=True` work better?

[Here](https://huggingface.co/docs/datasets/stream) is more about how and when to use it.

---

<div class="post-metadata">

**Author:** ![kopyl](https://avatars.discourse-cdn.com/v4/letter/k/3bc359/32.png) [@kopyl](https://discuss.huggingface.co/u/kopyl)\
**Post date:** [December 19, 2023, 4:31am UTC](https://discuss.huggingface.co/t/extremely-slow-data-loading-of-imagefolder/60131/4 "2023-12-19T04:31:09Z")

</div>

> [@panigrah](#):
>
> streaming=Tru

i’m basically loading it to upload to HF… So it still need to download it, right?

---

<div class="post-metadata">

**Author:** ![panigrah](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@panigrah](https://discuss.huggingface.co/u/panigrah)\
**Post date:** [December 19, 2023, 6:25am UTC](https://discuss.huggingface.co/t/extremely-slow-data-loading-of-imagefolder/60131/5 "2023-12-19T06:25:11Z")

</div>

if you are downloading only to upload it, then streaming will not work. [push\_to\_hub](https://github.com/huggingface/datasets/issues/3753) does not support streaming datasets.

---

<div class="post-metadata">

**Author:** ![kopyl](https://avatars.discourse-cdn.com/v4/letter/k/3bc359/32.png) [@kopyl](https://discuss.huggingface.co/u/kopyl)\
**Post date:** [December 19, 2023, 6:29am UTC](https://discuss.huggingface.co/t/extremely-slow-data-loading-of-imagefolder/60131/6 "2023-12-19T06:29:23Z")

</div>

@panigrah thank you very much. Maybe you also know if it’s possible to download a dataset in multi-processed way? For some reason [setting num\_proc](https://discuss.huggingface.co/t/how-can-i-multithreadedly-download-a-huggingface-dataset/56178/2) does not work at all… My dataset has 58 parquet files and i was hoping passing `num_proc` to `load_dataset` would spawn 58 Python processes each downloading its own parquet so I can load my dataset in 1 minutes instead of 50…

---

<div class="post-metadata">

**Author:** ![panigrah](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@panigrah](https://discuss.huggingface.co/u/panigrah)\
**Post date:** [December 19, 2023, 12:20pm UTC](https://discuss.huggingface.co/t/extremely-slow-data-loading-of-imagefolder/60131/7 "2023-12-19T12:20:26Z")

</div>

Where are you downloading from? I can try to replicate and see why

---

<div class="post-metadata">

**Author:** ![kopyl](https://avatars.discourse-cdn.com/v4/letter/k/3bc359/32.png) [@kopyl](https://discuss.huggingface.co/u/kopyl)\
**Post date:** [December 20, 2023, 2:47pm UTC](https://discuss.huggingface.co/t/extremely-slow-data-loading-of-imagefolder/60131/8 "2023-12-20T14:47:33Z")

</div>

From my private dataset. The structure is very similar to this one: [kopyl/833-icons-dataset-1024-blip-large · Datasets at Hugging Face](https://huggingface.co/datasets/kopyl/833-icons-dataset-1024-blip-large)

except that has 3 000 000 rows instead of 833

Thank you 🙂

---

<div class="post-metadata">

**Author:** ![francescorubbo](https://avatars.discourse-cdn.com/v4/letter/f/9de053/32.png) [@francescorubbo](https://discuss.huggingface.co/u/francescorubbo)\
**Post date:** [December 30, 2023, 3:35pm UTC](https://discuss.huggingface.co/t/extremely-slow-data-loading-of-imagefolder/60131/9 "2023-12-30T15:35:32Z")

</div>

To answer @sienna223’s original questions:

- the [FolderBasedBuilder](https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L32) doesn’t seem to be optimized for large datasets. It performs several costly operations parsing and validating the data.
- a simple [loading script](https://huggingface.co/docs/datasets/image_dataset#loading-script) sidesteps most of the costly operations, e.g.

```auto
import datasets
import glob

_IMAGES = glob.glob("my-image-folder/*.png")

class MyDataset(datasets.GeneratorBasedBuilder):
    """My image dataset"""

    def _info(self):
        return datasets.DatasetInfo(
            features=datasets.Features(
                {
                    "image": datasets.Image(),
                    # add labels or additional metadata as needed
                }
        ),
    )

    def _split_generators(self, dl_manager):
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN,
                gen_kwargs={
                    "images": _IMAGES,
                },
            ),
        ]

    def _generate_examples(self, images):
        for file_path in images:
            yield file_path, {
                "image": file_path,
                # parse labels or additional metadata as needed
            }

```

In my case, I was trying to create a 15M image dataset: the “imagefolder” approach seemed to run out of memory after 8 hours or so; with the loading script I was able to create the dataset in minutes.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [January 4, 2024, 5:49pm UTC](https://discuss.huggingface.co/t/extremely-slow-data-loading-of-imagefolder/60131/10 "2024-01-04T17:49:19Z")

</div>

We just released `datasets` 2.16.1 which optimizes the data files resolutions and makes it possible to load datasets with millions of images. It also requires `huggingface-hub` \>= 0.20.1 🙂

Older versions of `datasets` and `huggingface-hub` are slow to handle that many files