# Is there a size limit for dataset hosting

**URL:** <https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861>\
**Category:** 🤗Datasets\
**Created:** [February 18, 2022, 1:03pm UTC](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861 "2022-02-18T13:03:53Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![PaulLerner](https://avatars.discourse-cdn.com/v4/letter/p/65b543/32.png) [@PaulLerner](https://discuss.huggingface.co/u/PaulLerner)\
**Post date:** [February 18, 2022, 1:03pm UTC](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/1 "2022-02-18T13:03:53Z")

</div>

Hey,

I’d like to host a dataset on [Hugging Face – The AI community building the future.](https://huggingface.co/datasets/)  
Is there a size limit? My dataset is 106GB.  
I tried to add it but I can’t even add the files to the git repo (I’m getting `fatal: Out of memory, realloc failed` after `git add`), although I’m using `git-lfs`

Bests,

---

<div class="post-metadata">

**Author:** ![dynamicwebpaige](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dynamicwebpaige/32/4185_2.png) [@dynamicwebpaige](https://discuss.huggingface.co/u/dynamicwebpaige)\
**Post date:** [March 6, 2022, 5:48am UTC](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/2 "2022-03-06T05:48:31Z")

</div>

Hi, @PaulLerner! Thank you for the question. 🙂

Unfortunately, [`git-lfs` has a size limitation of 2GB](https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-git-large-file-storage#about-git-large-file-storage) – and only a few GB larger, if you’re using GitHub Enterprise. As an alternative, you can store your dataset in an alternate location ([ex: cloud storage](https://huggingface.co/docs/datasets/filesystems)), and reference that location in your data loading script.

---

<div class="post-metadata">

**Author:** ![osanseviero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/osanseviero/32/38738_2.png) [@osanseviero](https://discuss.huggingface.co/u/osanseviero)\
**Post date:** [March 6, 2022, 4:30pm UTC](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/3 "2022-03-06T16:30:04Z")

</div>

Hi @PaulLerner and @dynamicwebpaige!

As far as I know, we do have datasets with some Terabytes. As Paige suggested, you can store your dataset in alternate locations, but it is also possible (as far as I know) to upload datasets above 5GB with `huggingface-cli lfs-enable-largefiles `.

This is similar to the solution in [Uploading files larger than 5GB to model hub](https://discuss.huggingface.co/t/uploading-files-larger-than-5gb-to-model-hub/4081).

I hope this helps! 🚀 🌕

---

<div class="post-metadata">

**Author:** ![julien-c](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/julien-c/32/41937_2.png) [@julien-c](https://discuss.huggingface.co/u/julien-c)\
**Post date:** [March 7, 2022, 5:30pm UTC](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/4 "2022-03-07T17:30:38Z")

</div>

Yes unlike GitHub (😇😉) we do not have a limitation on LFS-stored file sizes on the HuggingFace hub.

However for ease of use and download speeds, we do advise users to chunk their large files into blobs of at most 20GB (or even 5GB if possible)

---

<div class="post-metadata">

**Author:** ![PaulLerner](https://avatars.discourse-cdn.com/v4/letter/p/65b543/32.png) [@PaulLerner](https://discuss.huggingface.co/u/PaulLerner)\
**Post date:** [March 10, 2022, 12:26pm UTC](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/5 "2022-03-10T12:26:07Z")

</div>

Hi everyone,

Thanks for your answers! `huggingface-cli lfs-enable-largefiles .` seems to do the job 🙂

---

<div class="post-metadata">

**Author:** ![wendlerc](https://avatars.discourse-cdn.com/v4/letter/w/919ad9/32.png) [@wendlerc](https://discuss.huggingface.co/u/wendlerc)\
**Post date:** [July 5, 2023, 11:33am UTC](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/7 "2023-07-05T11:33:06Z")

</div>

No limit for individual files. Ok wow that sounds great!!! Does this mean that there is also no limit for all files combined?

Best,  
Chris

---

<div class="post-metadata">

**Author:** ![lrauch](https://avatars.discourse-cdn.com/v4/letter/l/50afbb/32.png) [@lrauch](https://discuss.huggingface.co/u/lrauch)\
**Post date:** [July 26, 2023, 8:15am UTC](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/8 "2023-07-26T08:15:49Z")

</div>

I would also be interested in this question. I cannot find a clear limit for data hosting on the website.

---

<div class="post-metadata">

**Author:** ![severo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/severo/32/27449_2.png) [@severo](https://discuss.huggingface.co/u/severo)\
**Post date:** [July 26, 2023, 2:43pm UTC](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/9 "2023-07-26T14:43:17Z")

</div>

As said above ([Is there a size limit for dataset hosting - #4 by julien-c](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/4)), there is no fixed limit

---

<div class="post-metadata">

**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\
**Post date:** [August 21, 2023, 11:43pm UTC](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/10 "2023-08-21T23:43:45Z")

</div>

@julien-c I’m also confused. So I may upload as many files as I want and have them be \<20GB or \<5GB but I basically upload a data set as large as I want? @julien-c seems amazing…

---

<div class="post-metadata">

**Author:** ![severo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/severo/32/27449_2.png) [@severo](https://discuss.huggingface.co/u/severo)\
**Post date:** [August 23, 2023, 1:41pm UTC](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/12 "2023-08-23T13:41:00Z")

</div>

More details here: [Upload files to the Hub](https://huggingface.co/docs/huggingface_hub/main/en/guides/upload#tips-and-tricks-for-large-uploads)

The summary table:

| Characteristic | Recommended | Tips |
| --- | --- | --- |
| Repo size | - | contact us for large repos (TBs of data) |
| Files per repo | \<100k | merge data into fewer files |
| Entries per folder | \<10k | use subdirectories in repo |
| File size | \<5GB | split data into chunked files |
| Commit size | \<100 files\* | upload files in multiple commits |
| Commits per repo | - | upload multiple files per commit |

---

<div class="post-metadata">

**Author:** ![ProgramComputer](https://avatars.discourse-cdn.com/v4/letter/p/ecd19e/32.png) [@ProgramComputer](https://discuss.huggingface.co/u/ProgramComputer)\
**Post date:** [August 23, 2023, 6:10pm UTC](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/13 "2023-08-23T18:10:32Z")

</div>

@julien-c The limit as of now is 50 GB for each LFS file on upload. I did not see this in the documentation.

---

<div class="post-metadata">

**Author:** ![julien-c](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/julien-c/32/41937_2.png) [@julien-c](https://discuss.huggingface.co/u/julien-c)\
**Post date:** [August 24, 2023, 10:30am UTC](https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/14 "2023-08-24T10:30:57Z")

</div>

that’s correct. I think we should add it somewhere @severo