# Optimizing Disk Usage for Large (Audio) Datasets **URL:** \ **Category:** 🤗Datasets\ **Created:** [November 29, 2024, 2:17pm UTC](https://discuss.huggingface.co/t/optimizing-disk-usage-for-large-audio-datasets/128451 "2024-11-29T14:17:36Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![mwirth7](https://avatars.discourse-cdn.com/v4/letter/m/bbe5ce/32.png) [@mwirth7](https://discuss.huggingface.co/u/mwirth7)\ **Post date:** [November 29, 2024, 2:17pm UTC](https://discuss.huggingface.co/t/optimizing-disk-usage-for-large-audio-datasets/128451/1 "2024-11-29T14:17:36Z")
I’m part of the [BirdSet](https://huggingface.co/datasets/DBD-research-group/BirdSet) team, and we’ve identified an issue with our current [Builder script](https://huggingface.co/datasets/DBD-research-group/BirdSet/blob/main/BirdSet.py). Some of the audio datasets we work with are quite large, and we aim to provide access to individual audio files. To achieve this, we first download the archive file, extract its contents, and then generate the dataset. The reason for accessing the audio files directly is that we don’t need to load the entire audio file but only specific parts, which is possible using the `soundfile` library. This approach improves audio decoding efficiency and, consequently, reduces training time. However, we still want to provide access to the full audio files for other use cases. Problem: We noticed that this approach requires more than double the amount of disk space compared to the actual size of the files, as both the archives and the extracted audio files need to be stored simultaneously. We’ve implemented a workaround that achieves the same functionality, but it feels unintuitive and somewhat hacky. You can view this workaround [here](https://huggingface.co/datasets/DBD-research-group/BirdSet/blob/development/BirdSet.py). Essencially we don’t extract the archives anymore and during the generation of the Arrow files we load the audios bytes into the arrow files and delete archives as soon as they are not needed anymore. This way at every moment we only need half of the disk space we need at the moment. Is this a reasonable way to handle the problem, or are there alternative approaches we might not be aware of? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [November 29, 2024, 2:29pm UTC](https://discuss.huggingface.co/t/optimizing-disk-usage-for-large-audio-datasets/128451/2 "2024-11-29T14:29:59Z")
I’m not familiar with the datasets library, but I wonder if iter\_archive could be used? > > **[Create an audio dataset](https://huggingface.co/docs/datasets/audio_dataset)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ---
**Author:** ![mwirth7](https://avatars.discourse-cdn.com/v4/letter/m/bbe5ce/32.png) [@mwirth7](https://discuss.huggingface.co/u/mwirth7)\ **Post date:** [November 29, 2024, 2:43pm UTC](https://discuss.huggingface.co/t/optimizing-disk-usage-for-large-audio-datasets/128451/3 "2024-11-29T14:43:45Z")
Any suggestion is welcome. Good catch, but we are already using this, for [example](https://huggingface.co/datasets/DBD-research-group/BirdSet/blob/main/BirdSet.py#L447). I was testing earlier if i could extract and immediatly delete the archive but the `_generate_examples` function iterates through the archives content, so this doesn’t work with our current implementation. `iter_archive` seems to be the why that actually doesn’t work. With a clever rewrite this approach could possibly also work. But seems fairly far fetched and not aligned with other builder scripts in the audio domain. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [November 29, 2024, 2:51pm UTC](https://discuss.huggingface.co/t/optimizing-disk-usage-for-large-audio-datasets/128451/4 "2024-11-29T14:51:36Z")
I see. Then in this case you would be hard pressed to do without another library to manipulate tar.gz, but maybe you don’t want to add more dependencies… If you’re going to do it with just the standard Python library, datasets, and soundfile, I do think it’s going to be hacky… And you probably don’t want to have the means to get the dataset itself beforehand and put it in HF. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [November 30, 2024, 3:54pm UTC](https://discuss.huggingface.co/t/optimizing-disk-usage-for-large-audio-datasets/128451/5 "2024-11-30T15:54:31Z")
Hi ! if you `load_dataset()` a [AudioFolder](https://huggingface.co/docs/datasets/v3.1.0/en/audio_load#audiofolder) formatted dataset it won’t double the storage (the arrow table of the dataset will just contain links to the audio files on your disk) ---
**Author:** ![mwirth7](https://avatars.discourse-cdn.com/v4/letter/m/bbe5ce/32.png) [@mwirth7](https://discuss.huggingface.co/u/mwirth7)\ **Post date:** [December 2, 2024, 12:09pm UTC](https://discuss.huggingface.co/t/optimizing-disk-usage-for-large-audio-datasets/128451/6 "2024-12-02T12:09:09Z")
Hi! @lhoestq I believe this is what i am currently doing in the dataset builder [here](https://huggingface.co/datasets/DBD-research-group/BirdSet/blob/b0c14a03571a7d73d56b12c4b1db81952c4f7e64/BirdSet.py#L525). Instead of loading the audio using `audio.read()` which loads the bytes, i’m just passing the filepath string to the `datasets.Audio()` column. Here my current problem is that we have uploaded the audios in a `tar.gz` format which needs to be extracted to be able to pass the filepath string. Hence we [extract](https://huggingface.co/datasets/DBD-research-group/BirdSet/blob/b0c14a03571a7d73d56b12c4b1db81952c4f7e64/BirdSet.py#L440) the archive file. This way we both save the downloaded `tar.gz` and extracted audios, thus needing double the amount of storage. Currently i am looking how [minds14](https://huggingface.co/datasets/PolyAI/minds14/blob/main/minds14.py) handels their [extraction](https://huggingface.co/datasets/PolyAI/minds14/blob/75900a7c6f93f014f25b50d16596a6da89add3a5/minds14.py#L130) and `_generate_examples` function. Here it should be possible to sequentially extract all `tar.gz` files and delete them right after extraction. Would this be a good alternative? Or is there an easier way using AudioFolders? (Note: We have large amount of individual audio files) ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [December 2, 2024, 6:49pm UTC](https://discuss.huggingface.co/t/optimizing-disk-usage-for-large-audio-datasets/128451/7 "2024-12-02T18:49:21Z")
I’d recommend you to try AudioFolder or streaming WebDataset which are well optimized already and don’t duplicate the data locally