# Handling Large-Scale Image Dataset **URL:** \ **Category:** 🤗Datasets\ **Created:** [February 7, 2025, 11:33am UTC](https://discuss.huggingface.co/t/handling-large-scale-image-dataset/139489 "2025-02-07T11:33:44Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Ryoo72](https://avatars.discourse-cdn.com/v4/letter/r/3da27b/32.png) [@Ryoo72](https://discuss.huggingface.co/u/Ryoo72)\ **Post date:** [February 7, 2025, 11:33am UTC](https://discuss.huggingface.co/t/handling-large-scale-image-dataset/139489/1 "2025-02-07T11:33:44Z")
Hello. I want to train my VLM using a large-scale image dataset with the Huggingface trainer. I initially planned to follow the method suggested by [HuggingFace4M](https://huggingface.co/datasets/HuggingFaceH4/llava-instruct-mix-vsft), which involves embedding PIL images within arrow files. However, I found this approach problematic when dealing with datasets like DocStruct4M. The challenges included handling large files such as infographics and processing millions of data points. Furthermore, uploading such large image datasets to the hub proved difficult. Therefore, I’m curious about what would be the best way to structure a large-scale image dataset." ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [February 8, 2025, 6:48am UTC](https://discuss.huggingface.co/t/handling-large-scale-image-dataset/139489/2 "2025-02-08T06:48:18Z")
There seem to be several ways to create a dataset that does not involve uploading large files to the Hub. The method of writing a dataset loading script has been used for some time. A newer method is to use the Builder class, which I think is cleaner. However, with both of these methods, it is difficult to create a detailed structure as when everything is managed locally and uploaded… that’s just the way it is. ## Create a dataset loading script [https://huggingface.co/docs/datasets/dataset\_script](https://huggingface.co/docs/datasets/dataset_script) ### (in Japanese) > **[HuggingFace datasetsのBuilder classを使ってデータセットを自作する](https://zenn.dev/mjun0812/articles/cfe4b7346ba6b4)** ---
**Author:** ![Ryoo72](https://avatars.discourse-cdn.com/v4/letter/r/3da27b/32.png) [@Ryoo72](https://discuss.huggingface.co/u/Ryoo72)\ **Post date:** [February 8, 2025, 6:17pm UTC](https://discuss.huggingface.co/t/handling-large-scale-image-dataset/139489/3 "2025-02-08T18:17:55Z")
It seems we’re facing similar challenges. First of all, I really appreciate all the help you’ve provided with my problem. I was wondering if you could tell me whether a dataset created by subclassing `GeneratorBasedBuilder` uses a map-based or iterative approach? If possible, I’d like to use the map-based approach for faster processing speed, shuffling capability, and to know the total number of iterations. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [February 9, 2025, 3:43am UTC](https://discuss.huggingface.co/t/handling-large-scale-image-dataset/139489/4 "2025-02-09T03:43:24Z")
Hmm, we’ll have to ask the author to find out… @lhoestq ---
**Author:** ![Ryoo72](https://avatars.discourse-cdn.com/v4/letter/r/3da27b/32.png) [@Ryoo72](https://discuss.huggingface.co/u/Ryoo72)\ **Post date:** [February 9, 2025, 4:39am UTC](https://discuss.huggingface.co/t/handling-large-scale-image-dataset/139489/5 "2025-02-09T04:39:29Z")
Oh, that’s an actually great idea. ---
**Author:** ![Ryoo72](https://avatars.discourse-cdn.com/v4/letter/r/3da27b/32.png) [@Ryoo72](https://discuss.huggingface.co/u/Ryoo72)\ **Post date:** [February 9, 2025, 8:00am UTC](https://discuss.huggingface.co/t/handling-large-scale-image-dataset/139489/6 "2025-02-09T08:00:45Z")
hey mate, I think using arrow files to construct large image datasets isn’t a good idea. [Errror when saving to disk a dataset of images · Issue #5717 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/5717) It seems arrow has limitations in handling large image sets. I think storing image paths as strings might be the best solution. Have you tried using the Builder class? It seems to work in an iterative way, so I’m worried that since you don’t know the step length, it would be difficult to automatically schedule the learning rate and shuffling might not work as freely as we’d like. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [February 9, 2025, 8:32am UTC](https://discuss.huggingface.co/t/handling-large-scale-image-dataset/139489/7 "2025-02-09T08:32:03Z")
I see… In my case, the data set was small in terms of numbers. In that case, for large data sets, rather than handling them on a file-by-file basis, it might be faster to use a method like grouping the data sets into a certain number of sets using WebDataset, or, although it’s a bit primitive, dividing the dataset repo into multiple volumes (Volume 1, Volume 2, etc.) and manually merging them when necessary when loading. Whether it’s smart or not, that’s beside the point… > **[WebDataset](https://huggingface.co/docs/hub/datasets-webdataset)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science.