# How to get around rate limits?

**URL:** <https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947>\
**Category:** Beginners\
**Created:** [April 23, 2025, 10:28am UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947 "2025-04-23T10:28:26Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![mysocratesnote](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mysocratesnote/32/46167_2.png) [@mysocratesnote](https://discuss.huggingface.co/u/mysocratesnote)\
**Post date:** [April 23, 2025, 10:28am UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/1 "2025-04-23T10:28:26Z")

</div>

I’m trying to upload my first dataset. It’s only ~75,000 files and about 1GB but I was immediately I got 429 errors.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 23, 2025, 10:47am UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/2 "2025-04-23T10:47:47Z")

</div>

It is possible to mitigate this in Pro or Enterprise, but it may be quicker to reduce the number of requests.

For example, uploading with upload\_folder instead of upload\_file will result in fewer requests.

> [@Too many requests for URL](https://discuss.huggingface.co/t/too-many-requests-for-url/139743):
>
> I am using a finetuned llama model for inference (using vllm) and keep getting this error: Traceback (most recent call last): File "/pfss/mlde/workspaces/mlde\_wsp\_Rohrbach/users/ns94feza/.conda/envs/llmonk2/lib/python3.11/site-packages/huggingface\_hub/utils/\_http.py", line 406, in hf\_raise\_for\_status response.raise\_for\_status() File "/pfss/mlde/workspaces/mlde\_wsp\_Rohrbach/users/ns94feza/.conda/envs/llmonk2/lib/python3.11/site-packages/requests/models.py", line 1024, in raise\_for\_status…

> [@Getting 429 Error for sentence-transformers/all-mpnet-base-v2](https://discuss.huggingface.co/t/getting-429-error-for-sentence-transformers-all-mpnet-base-v2/107670):
>
> Hi Everyone, I faced this rate limit error for the first time for sentence-transformers. 429 Client Error: Too Many Requests for url: [https://api-inference.huggingface.co/pipeline/feature-extraction/sentence-transformers/all-mpnet-base-v2](https://api-inference.huggingface.co/pipeline/feature-extraction/sentence-transformers/all-mpnet-base-v2) Anyone else faced this? How we can avoid this error?

---

<div class="post-metadata">

**Author:** ![mysocratesnote](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mysocratesnote/32/46167_2.png) [@mysocratesnote](https://discuss.huggingface.co/u/mysocratesnote)\
**Post date:** [April 23, 2025, 10:57am UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/3 "2025-04-23T10:57:01Z")

</div>

I was using hugging-face-cli upload-large-folder

Is upload\_folder better?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 23, 2025, 11:13am UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/4 "2025-04-23T11:13:28Z")

</div>

The large version is still under development and is intended for cases where the size is truly large, so I think upload\_folder is better if the total size is within 50 GB.

---

<div class="post-metadata">

**Author:** ![mysocratesnote](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mysocratesnote/32/46167_2.png) [@mysocratesnote](https://discuss.huggingface.co/u/mysocratesnote)\
**Post date:** [April 23, 2025, 11:31am UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/5 "2025-04-23T11:31:00Z")

</div>

Does that mean:

`huggingface-cli upload_folder username/repository /path/to/dataset --repo-type=dataset --num-workers=12`

I’m getting that both upload\_folder and upload-folder are invalid arguments.

valid choices {download,upload,repo-files,env,login,whoami,logout,auth,repo,lfs-enable-largefiles,lfs-multipart-upload,scan-cache,delete-cache,tag,version,upload-large-folder}

Thanks.

---

<div class="post-metadata">

**Author:** ![mysocratesnote](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mysocratesnote/32/46167_2.png) [@mysocratesnote](https://discuss.huggingface.co/u/mysocratesnote)\
**Post date:** [April 23, 2025, 11:49am UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/6 "2025-04-23T11:49:01Z")

</div>

It works with

`huggingface-cli upload mysocratesnote/jfk-files-text ~/Desktop/extracted_text/releases --repo-type=dataset`

But it’s recommending I do it another way:

Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See [Environment variables](https://huggingface.co/docs/huggingface_hub/hf_transfer) for more details.

It seems you are trying to upload a large folder at once. This might take some time and then fail if the folder is too large. For such cases, it is recommended to upload in smaller batches or to use `HfApi().upload_large_folder(...)`/`huggingface-cli upload-large-folder` instead. For more details, check out [Upload files to the Hub](https://huggingface.co/docs/huggingface_hub/main/en/guides/upload#upload-a-large-folder).

Start hashing 73480 files.

Finished hashing 73480 files.

---

<div class="post-metadata">

**Author:** ![mysocratesnote](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mysocratesnote/32/46167_2.png) [@mysocratesnote](https://discuss.huggingface.co/u/mysocratesnote)\
**Post date:** [April 23, 2025, 12:03pm UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/7 "2025-04-23T12:03:33Z")

</div>

This failed shortly after it started with the ‘upload’ option.

File “/Users/user/miniforge3/lib/python3.12/site-packages/huggingface\_hub/utils/\_http.py”, line 409, in hf\_raise\_for\_status  
response.raise\_for\_status()  
File “/Users/user/miniforge3/lib/python3.12/site-packages/requests/models.py”, line 1024, in raise\_for\_status  
raise HTTPError(http\_error\_msg, response=self)  
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: [https://huggingface.co/api/datasets/mysocratesnote/jfk-files-text/commit/main](https://huggingface.co/api/datasets/mysocratesnote/jfk-files-text/commit/main)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):  
File “/Users/user/miniforge3/bin/huggingface-cli”, line 8, in   
sys.exit(main())  
^^^^^^  
File “/Users/user/miniforge3/lib/python3.12/site-packages/huggingface\_hub/commands/huggingface\_cli.py”, line 57, in main  
service.run()  
File “/Users/user/miniforge3/lib/python3.12/site-packages/huggingface\_hub/commands/upload.py”, line 206, in run  
print(self.\_upload())  
^^^^^^^^^^^^^^  
File “/Users/user/miniforge3/lib/python3.12/site-packages/huggingface\_hub/commands/upload.py”, line 301, in \_upload  
return self.api.upload\_folder(  
^^^^^^^^^^^^^^^^^^^^^^^  
File “/Users/user/miniforge3/lib/python3.12/site-packages/huggingface\_hub/utils/\_validators.py”, line 114, in \_inner\_fn  
return fn(\*args, \*\*kwargs)  
^^^^^^^^^^^^^^^^^^^  
File “/Users/user/miniforge3/lib/python3.12/site-packages/huggingface\_hub/hf\_api.py”, line 1624, in \_inner  
return fn(self, \*args, \*\*kwargs)  
^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/Users/user/miniforge3/lib/python3.12/site-packages/huggingface\_hub/hf\_api.py”, line 4934, in upload\_folder  
commit\_info = self.create\_commit(  
^^^^^^^^^^^^^^^^^^^  
File “/Users/user/miniforge3/lib/python3.12/site-packages/huggingface\_hub/utils/\_validators.py”, line 114, in \_inner\_fn  
return fn(\*args, \*\*kwargs)  
^^^^^^^^^^^^^^^^^^^  
File “/Users/user/miniforge3/lib/python3.12/site-packages/huggingface\_hub/hf\_api.py”, line 1624, in \_inner  
return fn(self, \*args, \*\*kwargs)  
^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/Users/user/miniforge3/lib/python3.12/site-packages/huggingface\_hub/hf\_api.py”, line 4285, in create\_commit  
hf\_raise\_for\_status(commit\_resp, endpoint\_name=“commit”)  
File “/Users/user/miniforge3/lib/python3.12/site-packages/huggingface\_hub/utils/\_http.py”, line 482, in hf\_raise\_for\_status  
raise \_format(HfHubHTTPError, str(e), response) from e  
huggingface\_hub.errors.HfHubHTTPError: 504 Server Error: Gateway Time-out for url: [https://huggingface.co/api/datasets/mysocratesnote/jfk-files-text/commit/main](https://huggingface.co/api/datasets/mysocratesnote/jfk-files-text/commit/main)

---

<div class="post-metadata">

**Author:** ![mysocratesnote](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mysocratesnote/32/46167_2.png) [@mysocratesnote](https://discuss.huggingface.co/u/mysocratesnote)\
**Post date:** [April 23, 2025, 12:09pm UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/8 "2025-04-23T12:09:57Z")

</div>

It works with upload-large-folder for a little while, but even with --num-workers=2 it quickly hits a rate limit again.

Is there no way to upload and specify a rate under the limit?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 23, 2025, 1:16pm UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/9 "2025-04-23T13:16:23Z")

</div>

That’s strange. I don’t think the numbers are high enough to cause an error… @Wauplin

> <https://github.com/huggingface/huggingface_hub/issues/2587>
>
> \### Describe the bug
> 
> Hi,
> I'm using the experimental \`upload-large-folder\` feat…ure. I'm trying to upload ~2000 files, but I'm running into a 429 error. Would it be possible to increase the rate limit when using the \`upload-large-folder\` feature?
> Thanks!
> 
> \### Reproduction
> 
> \_No response\_
> 
> \### Logs
> 
> \_No response\_
> 
> \### System info
> 
> \`\`\`shell
> \- huggingface\_hub version: 0.25.1
> \- Python version: 3.12.5
> \- Running in iPython ?: No
> \- Running in notebook ?: No
> \- Running in Google Colab ?: No
> \- Running in Google Colab Enterprise ?: No
> \- Has saved token ?: True
> \- Configured git credential helpers: store
> \- FastAI: N/A
> \- Tensorflow: N/A
> \- Torch: 2.4.1
> \- Jinja2: 3.1.4
> \- Graphviz: N/A
> \- keras: N/A
> \- Pydot: N/A
> \- Pillow: 10.4.0
> \- hf\_transfer: 0.1.8
> \- gradio: N/A
> \- tensorboard: N/A
> \- numpy: 1.26.4
> \- pydantic: N/A
> \- aiohttp: 3.10.8
> \- ENDPOINT: https://huggingface.co
> \- HF\_HUB\_OFFLINE: False
> \- HF\_HUB\_DISABLE\_TELEMETRY: False
> \- HF\_HUB\_DISABLE\_PROGRESS\_BARS: None
> \- HF\_HUB\_DISABLE\_SYMLINKS\_WARNING: False
> \- HF\_HUB\_DISABLE\_EXPERIMENTAL\_WARNING: False
> \- HF\_HUB\_DISABLE\_IMPLICIT\_TOKEN: False
> \- HF\_HUB\_ENABLE\_HF\_TRANSFER: False
> \- HF\_HUB\_ETAG\_TIMEOUT: 10
> \- HF\_HUB\_DOWNLOAD\_TIMEOUT: 10
> \`\`\`

> <https://github.com/huggingface/huggingface_hub/issues/2030>
>
> \### Describe the bug
> 
> I tried to use upload\_folder to upload my dataset with ~36…0 GB image frames, but kept failing with the following error:
> 
> \`huggingface\_hub.utils.\_errors.HfHubHTTPError: 413 Client Error: Payload Too Large for url\`
> 
> \### Steps to reproduce the bug
> 
> The code for reference:
> 
> For details regarding the \`frames\` folder structure, you can check \[TAO\](https://motchallenge.net/tao\_download.php)
> 
> \`\`\`python
> from huggingface\_hub import upload\_folder
> from huggingface\_hub import login
> 
> login()
> upload\_folder(
> folder\_path="TAO/frames",
> repo\_id="chengyenhsieh/TAO-Amodal",
> path\_in\_repo="frames",
> repo\_type="dataset",
> multi\_commits=True,
> multi\_commits\_verbose=True,
> )
> \`\`\`
> 
> \### Expected behavior
> 
> Image frames are successfully updated to my Huggingface Dataset \[Repo\](https://huggingface.co/datasets/chengyenhsieh/TAO-Amodal)
> 
> \### Environment info
> 
> datasets 2.17.0
> python 3.8

---

<div class="post-metadata">

**Author:** ![Wauplin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wauplin/32/40815_2.png) [@Wauplin](https://discuss.huggingface.co/u/Wauplin)\
**Post date:** [April 23, 2025, 1:31pm UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/10 "2025-04-23T13:31:27Z")

</div>

Not a server-side hardcoded limit no but might be some technical issues. We are working on [Fix dynamic commit size by maximizemaxwell · Pull Request #3016 · huggingface/huggingface\_hub · GitHub](https://github.com/huggingface/huggingface_hub/pull/3016) to allow dynamic commit sizes which should help mitigate the issue.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 23, 2025, 2:11pm UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/11 "2025-04-23T14:11:48Z")

</div>

Thank you!

---

<div class="post-metadata">

**Author:** ![mysocratesnote](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mysocratesnote/32/46167_2.png) [@mysocratesnote](https://discuss.huggingface.co/u/mysocratesnote)\
**Post date:** [April 23, 2025, 2:14pm UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/12 "2025-04-23T14:14:52Z")

</div>

Not sure I understand all of this. Is there a way to upload at the “right speed” to avoid getting blocked? I’m on my third or fourth attempt. I can only upload a few thousand files without getting blocked. Even with a single worker, the rate limit keeps triggering. This is not a huge archive, it’s just a little over 1 GB. It’s taken about 4 hours just to upload about 1/3 of it.

---

<div class="post-metadata">

**Author:** ![Wauplin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wauplin/32/40815_2.png) [@Wauplin](https://discuss.huggingface.co/u/Wauplin)\
**Post date:** [April 23, 2025, 3:43pm UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/13 "2025-04-23T15:43:56Z")

</div>

The problem is not the total size but the number of files (around 70k+ in total?). upload-large-folder was not meant for that at first (my bad, I designed it mostly to upload folders with hundreds of large files instead of folders with tens of thousands of small files). Result is that we are committing them by chunk of 50 which is making hundreds of commits which triggers the rate limit. [Fix dynamic commit size by maximizemaxwell · Pull Request #3016 · huggingface/huggingface\_hub · GitHub](https://github.com/huggingface/huggingface_hub/pull/3016) is meant as a good workaroudn for that but it’s not finished yet.

In the meantime I don’t have many suggestions except making the upload more manual (i.e. running `huggingface-cli upload` on subparts of the repo)

---

<div class="post-metadata">

**Author:** ![mysocratesnote](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mysocratesnote/32/46167_2.png) [@mysocratesnote](https://discuss.huggingface.co/u/mysocratesnote)\
**Post date:** [April 23, 2025, 3:45pm UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/14 "2025-04-23T15:45:34Z")

</div>

This problem may be exacerbated by the fact that huggingface-cli keeps trying over and over after it already hit a 429 error. Ideally it would quit after getting that error a couple of times.

---

<div class="post-metadata">

**Author:** ![Wauplin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wauplin/32/40815_2.png) [@Wauplin](https://discuss.huggingface.co/u/Wauplin)\
**Post date:** [April 23, 2025, 3:48pm UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/15 "2025-04-23T15:48:08Z")

</div>

Oooh, I did not notice that the files are been uploaded as regular markdown files. This means that all the data is stored in the git history, not using LFS files stored on S3. This is most certainly the culprit. Usually we try to avoid storing data as “raw” as it makes everything very slow. This is why git+LFS (and now git+xet) has been developed.

If that doesn’t make any sense to you, it basically means that the way files are stored on the repo is not optimized. I would recommend:

- create a new separate repo
- make sure the .md files are tracked as LFS (can be done by modifying the .gitattributes files [.gitattributes · mysocratesnote/jfk-files-text at main](https://huggingface.co/datasets/mysocratesnote/jfk-files-text/blob/main/.gitattributes))
- upload files subpart by subpart (around 250 by 250 is good)
- once everything is uploaded, delete the original repo and move the new one under the previous namespace

Very sorry about this situation but I think starting with a clean state is really needed here.

---

<div class="post-metadata">

**Author:** ![mysocratesnote](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mysocratesnote/32/46167_2.png) [@mysocratesnote](https://discuss.huggingface.co/u/mysocratesnote)\
**Post date:** [April 23, 2025, 3:49pm UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/16 "2025-04-23T15:49:17Z")

</div>

It seems like it failed even faster when using ‘upload’ rather than ‘upload-large-folder’

When you say run it on parts of the repo one by one… OK but how do I ensure it’s uploaded to the right path?

If the repository looks like this:

```auto
├── 2017/ # 2017 release
│ ├── part_1/ # 2017 part 1 
│ ├── part_2/ # 2017 part 2
│ ├── part_3/ # 2017 part 3
│ ├── part_4/ # 2017 part 4
│ └── part_5/ # 2017 part 5 (originally labeled "additional")
├── 2018/ # 2018 release
│ ├── part_1/ # 2018 part 1
│ └── part_2/ # 2018 part 2 

```

Do use huggingface-cli like this if I want to start with the 2017 subfolder?

huggingface-cli upload mysocratesnote/jfk-files-text/2017 ~/Desktop/extracted\_text/releases/2017 --repo-type=dataset

Thanks.

---

<div class="post-metadata">

**Author:** ![Wauplin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wauplin/32/40815_2.png) [@Wauplin](https://discuss.huggingface.co/u/Wauplin)\
**Post date:** [April 23, 2025, 3:50pm UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/17 "2025-04-23T15:50:41Z")

</div>

Other (better) solution is to store the data in a format that does not require to upload each file individually. Typically, it could .parquet files with columns like “date” “filename” “content” where each row is a markdown file. This way you will have only a few .parquet files to upload which will solve all of your problems. Also, it will enable the Dataset Studio for your repo.

---

<div class="post-metadata">

**Author:** ![mysocratesnote](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mysocratesnote/32/46167_2.png) [@mysocratesnote](https://discuss.huggingface.co/u/mysocratesnote)\
**Post date:** [April 23, 2025, 3:52pm UTC](https://discuss.huggingface.co/t/how-to-get-around-rate-limits/151947/19 "2025-04-23T15:52:32Z")

</div>

I think I figured that out. Thanks.