# HTTP 504: Gateway timeout error when pushing dataset

**URL:** <https://discuss.huggingface.co/t/http-504-gateway-timeout-error-when-pushing-dataset/15527>\
**Category:** 🤗Datasets\
**Created:** [March 9, 2022, 5:04am UTC](https://discuss.huggingface.co/t/http-504-gateway-timeout-error-when-pushing-dataset/15527 "2022-03-09T05:04:13Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![mohamed-illiyas](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@mohamed-illiyas](https://discuss.huggingface.co/u/mohamed-illiyas)\
**Post date:** [March 9, 2022, 5:04am UTC](https://discuss.huggingface.co/t/http-504-gateway-timeout-error-when-pushing-dataset/15527/1 "2022-03-09T05:04:13Z")

</div>

I am trying to push a large dataset with help of

`dataset.push_to_hub()`

While pushing, it gives HTTP error :504

requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: [https://huggingface.co/api/datasets/name/repo-name/upload/main/data/train2-00041-of-00064.parquet](https://huggingface.co/api/datasets/name/repo-name/upload/main/data/train2-00041-of-00064.parquet)

How to avoid this whenever pushing large datasets

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [March 11, 2022, 11:43am UTC](https://discuss.huggingface.co/t/http-504-gateway-timeout-error-when-pushing-dataset/15527/2 "2022-03-11T11:43:34Z")

</div>

Hi ! As discussed on GitHub, feel free to try again (the server might have had some issues).

We’re also adding a retry mechanism to workaround 504 errors: [Retry HfApi call inside push\_to\_hub when 504 error by albertvillanova · Pull Request #3886 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/pull/3886)

We’ll do a new release of `datasets` soon to include this 🙂

---

<div class="post-metadata">

**Author:** ![davidhhmack](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/davidhhmack/32/42414_2.png) [@davidhhmack](https://discuss.huggingface.co/u/davidhhmack)\
**Post date:** [March 3, 2025, 4:04pm UTC](https://discuss.huggingface.co/t/http-504-gateway-timeout-error-when-pushing-dataset/15527/3 "2025-03-03T16:04:23Z")

</div>

I’ve been getting this today (200GB dataset, push in python), just love to raise this is still a problem (and it’d be so great to be able to resume pushes since they can take a really long time)

Creating parquet from Arrow format: 100%|██████████████████████████████████████████████████| 48/48 [00:00\<00:00, 329.12ba/s]  
Uploading the dataset shards: 46%|██████████████████████▊ | 192/420 [6:08:02\<7:17:03, 115.01s/it]  
Traceback (most recent call last):  
File “/Users/dmackparty/dev/presto-py/app/pipelines/.venv/lib/python3.12/site-packages/huggingface\_hub/utils/\_http.py”, line 409, in hf\_raise\_for\_status  
response.raise\_for\_status()  
File “/Users/dmackparty/dev/presto-py/app/pipelines/.venv/lib/python3.12/site-packages/requests/models.py”, line 1024, in raise\_for\_status  
raise HTTPError(http\_error\_msg, response=self)  
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: [https://huggingface.co/api/complete\_multipart?uploadId=ak3yajyFaNpfIpmsLljywyUVb1sNay5D5GUGKAjJcM1h2GjHSHWLUuXNQ2.rxlPK9Mydu5w.5iCJ5P9SRIw4tbO2Gk9bXPRqPshRZlskHKa.tpGImodDyUjcU1yH92La&bucket=hf-hub-lfs-us-east-1&prefix=repos%2Fbe%2Feb%2Fbeebea985423ba0ccb6c7ef0c6925225a4dcd28c85e8c4db44607f457a00ce2d&expiration=Tue%2C+04+Mar+2025+06%3A40%3A19+GMT&signature=3fe1b5f59f4c97236b73dc288be78fd3d1802a8fe42fc10db5d3342a8854fd50](https://huggingface.co/api/complete_multipart?uploadId=ak3yajyFaNpfIpmsLljywyUVb1sNay5D5GUGKAjJcM1h2GjHSHWLUuXNQ2.rxlPK9Mydu5w.5iCJ5P9SRIw4tbO2Gk9bXPRqPshRZlskHKa.tpGImodDyUjcU1yH92La&bucket=hf-hub-lfs-us-east-1&prefix=repos%2Fbe%2Feb%2Fbeebea985423ba0ccb6c7ef0c6925225a4dcd28c85e8c4db44607f457a00ce2d&expiration=Tue%2C+04+Mar+2025+06%3A40%3A19+GMT&signature=3fe1b5f59f4c97236b73dc288be78fd3d1802a8fe42fc10db5d3342a8854fd50)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 3, 2025, 4:16pm UTC](https://discuss.huggingface.co/t/http-504-gateway-timeout-error-when-pushing-dataset/15527/4 "2025-03-03T16:16:26Z")

</div>

Possibly **.push\_to\_hub()** issue? @lhoestq

> [@504 Gateway Time-out when uploading large dataset to enterprise organization](https://discuss.huggingface.co/t/504-gateway-time-out-when-uploading-large-dataset-to-enterprise-organization/142173):
>
> Hi We recently bought an enterprise subscription in order to increase our data usage limits. However, it is no longer possible for me to push large datasets to the hub (500+ GB), as it eventually crashes with a “504 Server Error: Gateway Time-out for url: [https://huggingface.co/api/datasets/{organization}/{dataset}/preupload/main](https://huggingface.co/api/datasets/%7Borganization%7D/%7Bdataset%7D/preupload/main)” I don’t know if this have anything to do with your recent change in data usage limits policy. I see that you write that you have to contact you in order to upload d…

---

<div class="post-metadata">

**Author:** ![davidhhmack](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/davidhhmack/32/42414_2.png) [@davidhhmack](https://discuss.huggingface.co/u/davidhhmack)\
**Post date:** [March 3, 2025, 4:31pm UTC](https://discuss.huggingface.co/t/http-504-gateway-timeout-error-when-pushing-dataset/15527/5 "2025-03-03T16:31:00Z")

</div>

I’m considering writing code to

1. build and save the large dataset to local hdd
2. Try this to upload it without the fails [Upload files to the Hub](https://huggingface.co/docs/huggingface_hub/v0.14.1/en/guides/upload#upload-a-folder-by-chunks)

Good idea?

---

<div class="post-metadata">

**Author:** ![davidhhmack](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/davidhhmack/32/42414_2.png) [@davidhhmack](https://discuss.huggingface.co/u/davidhhmack)\
**Post date:** [March 3, 2025, 4:31pm UTC](https://discuss.huggingface.co/t/http-504-gateway-timeout-error-when-pushing-dataset/15527/6 "2025-03-03T16:31:58Z")

</div>

(note I’m calling push\_to\_hub on a DatasetDict, to an enterprise organization)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 3, 2025, 4:48pm UTC](https://discuss.huggingface.co/t/http-504-gateway-timeout-error-when-pushing-dataset/15527/7 "2025-03-03T16:48:22Z")

</div>

> Good idea?

Yea. In that case, you could use this.

> **[HfApi Client](https://huggingface.co/docs/huggingface_hub/v0.29.1/en/package_reference/hf_api#huggingface_hub.HfApi.upload_large_folder)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [March 3, 2025, 6:11pm UTC](https://discuss.huggingface.co/t/http-504-gateway-timeout-error-when-pushing-dataset/15527/8 "2025-03-03T18:11:38Z")

</div>

cc @Wauplin maybe a transient 504 issue ?

Anyway @davidhhmack making push\_to\_hub easier to resume is definitely something we’ll look into !

---

<div class="post-metadata">

**Author:** ![davidhhmack](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/davidhhmack/32/42414_2.png) [@davidhhmack](https://discuss.huggingface.co/u/davidhhmack)\
**Post date:** [March 3, 2025, 6:27pm UTC](https://discuss.huggingface.co/t/http-504-gateway-timeout-error-when-pushing-dataset/15527/9 "2025-03-03T18:27:43Z")

</div>

Thanks, I appreciate it! It’s crushing to get hours through an upload, then have it die