# Duplicated cache- arrow files when uploading large folder?

**URL:** <https://discuss.huggingface.co/t/duplicated-cache-arrow-files-when-uploading-large-folder/147817>\
**Category:** 🤗Datasets\
**Created:** [March 28, 2025, 11:37am UTC](https://discuss.huggingface.co/t/duplicated-cache-arrow-files-when-uploading-large-folder/147817 "2025-03-28T11:37:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![samirchar](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@samirchar](https://discuss.huggingface.co/u/samirchar)\
**Post date:** [March 28, 2025, 11:37am UTC](https://discuss.huggingface.co/t/duplicated-cache-arrow-files-when-uploading-large-folder/147817/1 "2025-03-28T11:37:33Z")

</div>

Hello,

I have a folder with .arrow files that I previously created using save\_to\_disk. When I now use upload\_large\_folder I see some .cache- files being pushed to the hub, that I don’t see in the directory i’m pushing in my local machine. Is this normal? Are they duplicates or is HF splitting the file into two?

It seems that all files from my local folder were uploaded and these are additional

This is an example of the “extra” files:

[‘uniref50\_202401/arrow/train/cache-5438b1d15cbf9f5a\_00004\_of\_00024.arrow’, ‘uniref50\_202401/arrow/train/cache-77dd2d54eba47e69\_00004\_of\_00024.arrow’]

Can i just delete those in the repo?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 28, 2025, 12:25pm UTC](https://discuss.huggingface.co/t/duplicated-cache-arrow-files-when-uploading-large-folder/147817/2 "2025-03-28T12:25:08Z")

</div>

It seems like it’s okay to delete it, but if you’re worried, call lhonestq.

> [@How can I clean the dataset cache?](https://discuss.huggingface.co/t/how-can-i-clean-the-dataset-cache/73033/2):
>
> Feel free to delete the download directory. Then all the other directories are named after the datasets, and contain the cache files when you do load\_dataset() and to reload cached map() results

> **[Cache management](https://huggingface.co/docs/datasets/v3.5.0/cache)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [April 7, 2025, 10:38am UTC](https://discuss.huggingface.co/t/duplicated-cache-arrow-files-when-uploading-large-folder/147817/3 "2025-04-07T10:38:14Z")

</div>

cache files are the cache results of e.g. `map()` operations on the original data, you can delete them if you don’t need the `map()` results anymore 😉