# Image dataset best practices?

**URL:** <https://discuss.huggingface.co/t/image-dataset-best-practices/13974>\
**Category:** 🤗Datasets\
**Created:** [January 21, 2022, 11:54am UTC](https://discuss.huggingface.co/t/image-dataset-best-practices/13974 "2022-01-21T11:54:09Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![segments-bert](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/segments-bert/32/21889_2.png) [@segments-bert](https://discuss.huggingface.co/u/segments-bert)\
**Post date:** [January 21, 2022, 11:54am UTC](https://discuss.huggingface.co/t/image-dataset-best-practices/13974/1 "2022-01-21T11:54:09Z")

</div>

Hi! I’m one of the founders of [Segments.ai](https://segments.ai), a data labeling platform for computer vision. We’re working on an integration with HuggingFace, making it possible to export labeled datasets to the 🤗 hub.

From reading the docs and toying around a bit, I found there’s a few potential ways to set up an image dataset:

1. Keep the image files out of the repository, and download them from their URLs (they’re hosted in the cloud) in the dataset loading script. The disadvantage here is that if the image URLs ever become unavailable, the dataset also won’t work anymore.
2. Store the image files in the repository, _packed together in a few large parquet files_, using git-lfs. This is basically what happens when you create a dataset with an image column locally, and run `dataset.push_to_hub()`. See [this dataset](https://huggingface.co/datasets/segments-bert/pets/tree/main).
3. Store the image files in the repository _as individual jpg/png files_, using git-lfs. Compared to the previous approach, this requires a custom dataset loading script. This seems cleaner from a versioning point of view: when images are added or removed later on, it leads to a compact diff compared to working with the parquet files. But perhaps it’s not ideal to have so many small files in a git-lfs repo.

Do you have any recommendations on what would be the cleanest approach that is considered best practice?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [January 21, 2022, 3:48pm UTC](https://discuss.huggingface.co/t/image-dataset-best-practices/13974/2 "2022-01-21T15:48:40Z")

</div>

Hi Bert, thanks for reaching out, and good job with segments.ai !

You mentioned three different ways of hosting an image dataset, and they’re all sensible ideas. Here are a few aspects that can help deciding which one is best depending on your case:

1. Storing the URLs. It has several disadvantages: less convenient, less reproducibility, and probably doesn’t work in the long run. This should be avoided as much as possible IMO. However for certain datasets with copyright/licensing issues this can still be a solution, in particular if you’re not allowed to redistribute the images yourself.

2. Use Parquet files (e.g. from `push_to_hub()` or [pandas](https://huggingface.co/docs/hub/en/datasets-pandas)). It’s nice on several aspects:

3. Store the raw images. It Is very flexible since you can add/update/remove images pretty easily. It can be convenient especially for small datasets, however:

To conclude on this:

Option 2. Is the go-to solution if your dataset is big, frozen or if you need fancy stuff like parallel processing or streaming.

Option 3. Is preferable only for a small dataset, when constructing it or when you need flexibility

Option 1. Should be avoided, unless you have no other option

EDIT from 2024: another option is WebDataset, see [WebDataset](https://huggingface.co/docs/hub/en/datasets-webdataset)

cc @osanseviero

---

<div class="post-metadata">

**Author:** ![segments-tobias](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/segments-tobias/32/5052_2.png) [@segments-tobias](https://discuss.huggingface.co/u/segments-tobias)\
**Post date:** [January 21, 2022, 4:51pm UTC](https://discuss.huggingface.co/t/image-dataset-best-practices/13974/3 "2022-01-21T16:51:37Z")

</div>

Interesting!

Maybe it would be a good idea to convert the HF example datasets to Parquet files. I was trying to set up my own HF dataset and simply copied the approach of loading the images from URLs from the [mnist](https://github.com/huggingface/datasets/blob/master/datasets/mnist/mnist.py) dataset.

---

<div class="post-metadata">

**Author:** ![segments-bert](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/segments-bert/32/21889_2.png) [@segments-bert](https://discuss.huggingface.co/u/segments-bert)\
**Post date:** [January 21, 2022, 4:55pm UTC](https://discuss.huggingface.co/t/image-dataset-best-practices/13974/4 "2022-01-21T16:55:09Z")

</div>

Thanks Quentin for your extensive answer, very informative and extremely helpful!

Sounds like option 2 is the way to go for us, and is also the easiest to get started thanks to the `push_to_hub` functionality.

Expect some more dataset-related questions from us in the coming days ;).

---

<div class="post-metadata">

**Author:** ![osanseviero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/osanseviero/32/38738_2.png) [@osanseviero](https://discuss.huggingface.co/u/osanseviero)\
**Post date:** [January 21, 2022, 5:50pm UTC](https://discuss.huggingface.co/t/image-dataset-best-practices/13974/5 "2022-01-21T17:50:38Z")

</div>

FYI, the endpoints are public and are documented [here](https://huggingface.co/docs/hub/endpoints). That means that if you want to manage it through other programming languages it should be feasible as well.

There is also `huggingface_hub` ([huggingface\_hub/src/huggingface\_hub at main · huggingface/huggingface\_hub · GitHub](https://github.com/huggingface/huggingface_hub/tree/main/src/huggingface_hub)), a Python library to interact with these endpoints, but if you’re already using `push_to_hub`, that’s the way to go.

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [February 10, 2022, 2:54pm UTC](https://discuss.huggingface.co/t/image-dataset-best-practices/13974/6 "2022-02-10T14:54:26Z")

</div>

Came across a nice [blog post](https://danielvanstrien.xyz/metadata/deployment/huggingface/ethics/huggingface-datasets/faiss/2022/01/13/image_search.html) regarding uploading an image dataset to the hub using `push_to_hub` (option 2 in this thread).

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [February 14, 2022, 6:15pm UTC](https://discuss.huggingface.co/t/image-dataset-best-practices/13974/7 "2022-02-14T18:15:44Z")

</div>

Hi @segments-bert @segments-tobias ! We’ve just merged a [PR](https://github.com/huggingface/datasets/pull/3685) that adds support for the `Image` feature in `push_to_hub`. You can test it by installing `datasets` from master:

```auto
pip install git+https://github.com/huggingface/datasets

```

Any feedback is greatly appreciated!

---

<div class="post-metadata">

**Author:** ![segments-bert](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/segments-bert/32/21889_2.png) [@segments-bert](https://discuss.huggingface.co/u/segments-bert)\
**Post date:** [February 16, 2022, 3:16pm UTC](https://discuss.huggingface.co/t/image-dataset-best-practices/13974/8 "2022-02-16T15:16:03Z")

</div>

Hey @mariosasko , this is great! Before, I used the `dataset.map()` approach described in the blogpost mentioned by @nielsr. Now I can skip that and just push to the hub like this:

```auto
features = datasets.Features({
    'name': datasets.Value('string'),
    'image': datasets.Image(),
})

dataset_dict = {
    'name': ['cat.jpg'],
    'image': ['cat.jpg']
}

dataset = datasets.Dataset.from_dict(dataset_dict, features)
dataset.push_to_hub('segments-bert/image-upload-test')

```

A few additional questions:

- There is no difference between the `dataset.map()` approach and this approach, in how the images end up in the parquet files, right?
- Do I understand correctly that the image bytes are stored in decoded form in the parquet files, i.e. not in their original .jpg or .png format? I assume so, given [this comment](https://github.com/huggingface/datasets/pull/3685#pullrequestreview-876595269) by @lhoestq .
- Would it make sense to add a generic “File” feature, besides the existing Image and Audio features? I’m asking because our data labeling platform also supports video and point cloud labeling, for which the data comes in a variety of file formats (.pcd, .bin, .mp4, …). Would be nice to push such datasets to the hub as well.

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [February 16, 2022, 3:36pm UTC](https://discuss.huggingface.co/t/image-dataset-best-practices/13974/9 "2022-02-16T15:36:01Z")

</div>

Hi @segments-bert! Thanks, glad you like it.

> - There is no difference between the `dataset.map()` approach and this approach, in how the images end up in the parquet files, right?

Yes, there is no difference.

> - Do I understand correctly that the image bytes are stored in decoded form in the parquet files, i.e. not in their original .jpg or .png format? I assume so, given [this comment](https://github.com/huggingface/datasets/pull/3685#pullrequestreview-876595269) by @lhoestq .

The image bytes stored in the parquet files are raw/unprocessed.

> - Would it make sense to add a generic “File” feature, besides the existing Image and Audio features? I’m asking because our data labeling platform also supports video and point cloud labeling, for which the data comes in a variety of file formats (.pcd, .bin, .mp4, …). Would be nice to push such datasets to the hub as well.

An interesting proposal. We will consider it. For now, you can embed file bytes (`Value("binary")`) in a dataset before `push_to_hub` and apply some additional preprocessing after loading.

---

<div class="post-metadata">

**Author:** ![fulldecent](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fulldecent/32/13581_2.png) [@fulldecent](https://discuss.huggingface.co/u/fulldecent)\
**Post date:** [January 15, 2023, 11:43pm UTC](https://discuss.huggingface.co/t/image-dataset-best-practices/13974/10 "2023-01-15T23:43:44Z")

</div>

In choosing between option 2 and 3.

The images are being stored, so there is already no risk of data loss. That could be a major reason for having to remove a point from the data set. Since that is gone, it is very likely the data set will be append-only for the life of the project. So for this, there is not much value in using individual file with their more readable diffs and delta compression.