# Stream image dataset from (Azure) cloud storage

**URL:** <https://discuss.huggingface.co/t/stream-image-dataset-from-azure-cloud-storage/49442>\
**Category:** 🤗Datasets\
**Created:** [August 4, 2023, 7:30am UTC](https://discuss.huggingface.co/t/stream-image-dataset-from-azure-cloud-storage/49442 "2023-08-04T07:30:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sonnehansen](https://avatars.discourse-cdn.com/v4/letter/s/ed8c4c/32.png) [@sonnehansen](https://discuss.huggingface.co/u/sonnehansen)\
**Post date:** [August 4, 2023, 7:30am UTC](https://discuss.huggingface.co/t/stream-image-dataset-from-azure-cloud-storage/49442/1 "2023-08-04T07:30:49Z")

</div>

Hi,  
I am trying to make our image data into a HF-formatted dataset with custom data loader script stored on our Azure blob storage. I would then like to stream (or use fsspec / adlfs) to consume the dataset without downloading the full dataset.

I have looked around and found:

- official datasets doc on cloud storage, and I can make it work with `load_from_disk` but that is not what I want
- A [forum thread](https://discuss.huggingface.co/t/how-to-use-s3-path-with-load-dataset-with-streaming-true/25739/11) that discuss `load_dataset('s3:...', streaming=True)` and a related [github issue](https://github.com/huggingface/datasets/issues/5281) that is fixed by PR

(I apologize for the missing links to official docs and PR but being new to the forum I can only include 2 links in a post 😢 )

However, I still cannot make it work with image files (there is no documentation around the feature and most examples I find are the supported formats ‘json’, …, not image files), a custom data loader script and without passing through a step of downloading and or zipping all files (I managed to stream using the azure blob storage REST API and images in a zip file).

The reason I would prefer not to ZIP is because we consume images from the storage directly. I would prefer to use the same location for the dataset, rather than duplicating the data to another location. I am very open to reasons why I am wrong in seeing this as an undesirable pattern.

Any thoughts if what I want to achieve is possible and if so, how to go about it? Or alternatively, other best-practices that I should rather pursue.

Thanks in advance 🙂 , Mikael

---

<div class="post-metadata">

**Author:** ![vazgbruno](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vazgbruno/32/22248_2.png) [@vazgbruno](https://discuss.huggingface.co/u/vazgbruno)\
**Post date:** [December 27, 2023, 5:12pm UTC](https://discuss.huggingface.co/t/stream-image-dataset-from-azure-cloud-storage/49442/2 "2023-12-27T17:12:20Z")

</div>

I think I have a similar problem, but with audio files. Were you able to find a solution?

---

<div class="post-metadata">

**Author:** ![sonnehansen](https://avatars.discourse-cdn.com/v4/letter/s/ed8c4c/32.png) [@sonnehansen](https://discuss.huggingface.co/u/sonnehansen)\
**Post date:** [January 8, 2024, 2:33pm UTC](https://discuss.huggingface.co/t/stream-image-dataset-from-azure-cloud-storage/49442/3 "2024-01-08T14:33:42Z")

</div>

Hi @vazgbruno,

No, I was told it was [not supported](https://github.com/huggingface/datasets/issues/5281#issuecomment-1681691402) as of yet and I abandoned HF for the time being. It seems that the discussion continuous on the github issue.

Cheers  
Mikael

---

<div class="post-metadata">

**Author:** ![vazgbruno](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vazgbruno/32/22248_2.png) [@vazgbruno](https://discuss.huggingface.co/u/vazgbruno)\
**Post date:** [January 8, 2024, 2:45pm UTC](https://discuss.huggingface.co/t/stream-image-dataset-from-azure-cloud-storage/49442/4 "2024-01-08T14:45:55Z")

</div>

Thanks for answering and for sharing the GitHub issue @sonnehansen!

All the best,  
Bruno