# Problem pushing dataset to huggingface **URL:** \ **Category:** 🤗Datasets\ **Created:** [November 30, 2022, 9:45am UTC](https://discuss.huggingface.co/t/problem-pushing-dataset-to-huggingface/26993 "2022-11-30T09:45:30Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![fathyshalab](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fathyshalab/32/15975_2.png) [@fathyshalab](https://discuss.huggingface.co/u/fathyshalab)\ **Post date:** [November 30, 2022, 9:45am UTC](https://discuss.huggingface.co/t/problem-pushing-dataset-to-huggingface/26993/1 "2022-11-30T09:45:30Z")
I’m trying to push my dataset to the hub using the dataset.push\_to\_hub but get the following error: Pushing split train to the Hub. Domain: work Pushing dataset shards to the dataset hub: 0%| | 0/1 [00:00\ 8 dataset.push\_to\_hub(f"fathyshalab/{domain}“) **10** domain\_datasets[“work”] File ~/.conda/envs/baselines-transformers/lib/python3.9/site-packages/datasets/dataset\_dict.py:1350, in DatasetDict.push\_to\_hub(self, repo\_id, private, token, branch, max\_shard\_size, shard\_size, embed\_external\_files) **1348** logger.warning(f"Pushing split {split} to the Hub.”) **1349** # The split=key needs to be removed before merging → 1350 repo\_id, split, uploaded\_size, dataset\_nbytes, \_, \_ = self[split].\_push\_parquet\_shards\_to\_hub( **1351** repo\_id, **1352** split=split, **1353** private=private, **1354** token=token, **1355** branch=branch, **1356** max\_shard\_size=max\_shard\_size, **1357** embed\_external\_files=embed\_external\_files, **1358** ) **1359** total\_uploaded\_size += uploaded\_size **1360** total\_dataset\_nbytes += dataset\_nbytes File ~/.conda/envs/baselines-transformers/lib/python3.9/site-packages/datasets/arrow\_dataset.py:4195, in Dataset.\_push\_parquet\_shards\_to\_hub(self, repo\_id, split, private, token, branch, max\_shard\_size, embed\_external\_files) **4193** shard.to\_parquet(buffer) … **121** fn\_name=fn. **name** , has\_token=has\_token, kwargs=kwargs **122** ) → 124 return fn(\*args, \*\*kwargs) TypeError: upload\_file() got an unexpected keyword argument ‘identical\_ok’ ` What I understand that in the dataset\_arrow.py there is a identical\_ok argument that isnt used anymore or added but not included in the upload files function right? ` ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [December 1, 2022, 2:47pm UTC](https://discuss.huggingface.co/t/problem-pushing-dataset-to-huggingface/26993/2 "2022-12-01T14:47:32Z")
Can you try to update `datasets` and `huggingface-hub` ? ```python pip install -U datasets huggingface-hub ``` ---
**Author:** ![zmao](https://avatars.discourse-cdn.com/v4/letter/z/43a26b/32.png) [@zmao](https://discuss.huggingface.co/u/zmao)\ **Post date:** [December 9, 2022, 1:38am UTC](https://discuss.huggingface.co/t/problem-pushing-dataset-to-huggingface/26993/3 "2022-12-09T01:38:57Z")
I had same issue. Did you resolve it? I am pushing a image dataset with captions. ---
**Author:** ![eaqui](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/eaqui/32/13235_2.png) [@eaqui](https://discuss.huggingface.co/u/eaqui)\ **Post date:** [December 27, 2022, 2:18pm UTC](https://discuss.huggingface.co/t/problem-pushing-dataset-to-huggingface/26993/4 "2022-12-27T14:18:18Z")
Same issue happening to me when pushing the dataset. Using the same code for my tokenizer uploads correctly. Kindly help, details below: ```auto tokenizer.push_to_hub(repo_name) timit['train'].push_to_hub(repo_name) ``` - Name: huggingface-hub Version: 0.11.1 - Name: transformers Version: 4.25.1 - Name: datasets Version: 2.8.0 ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [January 3, 2023, 10:46am UTC](https://discuss.huggingface.co/t/problem-pushing-dataset-to-huggingface/26993/5 "2023-01-03T10:46:21Z")
Can you share the full stack trace please ? ---
**Author:** ![Kristinabckr](https://avatars.discourse-cdn.com/v4/letter/k/3ec8ea/32.png) [@Kristinabckr](https://discuss.huggingface.co/u/Kristinabckr)\ **Post date:** [January 3, 2023, 6:09pm UTC](https://discuss.huggingface.co/t/problem-pushing-dataset-to-huggingface/26993/7 "2023-01-03T18:09:50Z")
Hello @fathyshalab, I had the same issue when working with an older Common Voice dataset version. With updating the following versions and taking a newer Common Voice dataset, the problem was solved. Maybe you try this out: huggingface hub version 0.11.1 transformers version 4.25.1 datasets version 2.8.0 (and, as in the Common Voice case, not “common\_voice” but “mozilla-foundation/common\_voice\_11\_0”) Hope, this helps. ---
**Author:** ![Kamaljp](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kamaljp/32/15013_2.png) [@Kamaljp](https://discuss.huggingface.co/u/Kamaljp)\ **Post date:** [June 11, 2023, 9:24am UTC](https://discuss.huggingface.co/t/problem-pushing-dataset-to-huggingface/26993/8 "2023-06-11T09:24:05Z")
@lhoestq I am facing the same issue and the stack trace is as below ```auto --------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[29], line 1 ----> 1 medium_datasets.push_to_hub("Kamaljp/medium_articles") File /opt/conda/lib/python3.10/site-packages/datasets/dataset_dict.py:899, in DatasetDict.push_to_hub(self, repo_id, private, token, branch, shard_size, embed_external_files) 897 logger.warning(f"Pushing split {split} to the Hub.") 898 # The split=key needs to be removed before merging --> 899 repo_id, split, uploaded_size, dataset_nbytes = self[split]._push_parquet_shards_to_hub( 900 repo_id, 901 split=split, 902 private=private, 903 token=token, 904 branch=branch, 905 shard_size=shard_size, 906 embed_external_files=embed_external_files, 907 ) 908 total_uploaded_size += uploaded_size 909 total_dataset_nbytes += dataset_nbytes File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:3474, in Dataset._push_parquet_shards_to_hub(self, repo_id, split, private, token, branch, shard_size, embed_external_files) 3472 shard.to_parquet(buffer) 3473 uploaded_size += buffer.tell() -> 3474 _retry( 3475 api.upload_file, 3476 func_kwargs=dict( 3477 path_or_fileobj=buffer.getvalue(), 3478 path_in_repo=path_in_repo(index), 3479 repo_id=repo_id, 3480 token=token, 3481 repo_type="dataset", 3482 revision=branch, 3483 identical_ok=True, 3484 ), 3485 exceptions=HTTPError, 3486 status_codes=[504], 3487 base_wait_time=2.0, 3488 max_retries=5, 3489 max_wait_time=20.0, 3490 ) 3491 return repo_id, split, uploaded_size, dataset_nbytes File /opt/conda/lib/python3.10/site-packages/datasets/utils/file_utils.py:330, in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time) 328 while True: 329 try: --> 330 return func(*func_args, **func_kwargs) 331 except exceptions as err: 332 if retry >= max_retries or (status_codes and err.response.status_code not in status_codes): File /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:120, in validate_hf_hub_args.._inner_fn(*args, **kwargs) 117 if check_use_auth_token: 118 kwargs = smoothly_deprecate_use_auth_token(fn_name=fn. __name__ , has_token=has_token, kwargs=kwargs) --> 120 return fn(*args, **kwargs) TypeError: HfApi.upload_file() got an unexpected keyword argument 'identical_ok' ``` I am trying to push the data from Kaggle notebook to Hugging face hub, the above error is occuring. There is internet enabled in the notebook. I have been pushing from Colab Notebooks, it is working without any challenge. Any direction would be appreciated. Thanks ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [June 12, 2023, 9:38am UTC](https://discuss.huggingface.co/t/problem-pushing-dataset-to-huggingface/26993/9 "2023-06-12T09:38:06Z")
Have you tried updating `datasets` and `huggingface-hub` as suggested earlier in the thread ? ---
**Author:** ![Kamaljp](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kamaljp/32/15013_2.png) [@Kamaljp](https://discuss.huggingface.co/u/Kamaljp)\ **Post date:** [June 25, 2023, 3:08am UTC](https://discuss.huggingface.co/t/problem-pushing-dataset-to-huggingface/26993/10 "2023-06-25T03:08:14Z")
I am facing the same issue. Here are the details Environment: Kaggle Notebook libraries : pip install datasets transformers=4.28.1 Stack Trace: ```auto TypeError Traceback (most recent call last) Cell In[23], line 1 ----> 1 scraped_ds.push_to_hub("my_data_repo") File /opt/conda/lib/python3.10/site-packages/datasets/dataset_dict.py:899, in DatasetDict.push_to_hub(self, repo_id, private, token, branch, shard_size, embed_external_files) 897 logger.warning(f"Pushing split {split} to the Hub.") 898 # The split=key needs to be removed before merging --> 899 repo_id, split, uploaded_size, dataset_nbytes = self[split]._push_parquet_shards_to_hub( 900 repo_id, 901 split=split, 902 private=private, 903 token=token, 904 branch=branch, 905 shard_size=shard_size, 906 embed_external_files=embed_external_files, 907 ) 908 total_uploaded_size += uploaded_size 909 total_dataset_nbytes += dataset_nbytes File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:3474, in Dataset._push_parquet_shards_to_hub(self, repo_id, split, private, token, branch, shard_size, embed_external_files) 3472 shard.to_parquet(buffer) 3473 uploaded_size += buffer.tell() -> 3474 _retry( 3475 api.upload_file, 3476 func_kwargs=dict( 3477 path_or_fileobj=buffer.getvalue(), 3478 path_in_repo=path_in_repo(index), 3479 repo_id=repo_id, 3480 token=token, 3481 repo_type="dataset", 3482 revision=branch, 3483 identical_ok=True, 3484 ), 3485 exceptions=HTTPError, 3486 status_codes=[504], 3487 base_wait_time=2.0, 3488 max_retries=5, 3489 max_wait_time=20.0, 3490 ) 3491 return repo_id, split, uploaded_size, dataset_nbytes File /opt/conda/lib/python3.10/site-packages/datasets/utils/file_utils.py:330, in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time) 328 while True: 329 try: --> 330 return func(*func_args, **func_kwargs) 331 except exceptions as err: 332 if retry >= max_retries or (status_codes and err.response.status_code not in status_codes): File /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:118, in validate_hf_hub_args.._inner_fn(*args, **kwargs) 115 if check_use_auth_token: 116 kwargs = smoothly_deprecate_use_auth_token(fn_name=fn. __name__ , has_token=has_token, kwargs=kwargs) --> 118 return fn(*args, **kwargs) File /opt/conda/lib/python3.10/site-packages/huggingface_hub/hf_api.py:826, in future_compatible.._inner(self, *args, **kwargs) 823 return self.run_as_future(fn, self, *args, **kwargs) 825 # Otherwise, call the function normally --> 826 return fn(self, *args, **kwargs) TypeError: HfApi.upload_file() got an unexpected keyword argument 'identical_ok' ``` BG: I have repo with same name, and I have pushed the raw data to it already. I thought the above error occurs when I am pushing the data to the repo which has data which is identical to the “the new data” I am pushing. I then changed the repo\_name. Even after that the error persists. Kindly update what other data you need to help resolve the issue ---
**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\ **Post date:** [June 26, 2023, 10:01am UTC](https://discuss.huggingface.co/t/problem-pushing-dataset-to-huggingface/26993/11 "2023-06-26T10:01:57Z")
You can fix this by updating the datasets installation in the environment to the newest version (2.13). ---
**Author:** ![Kamaljp](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kamaljp/32/15013_2.png) [@Kamaljp](https://discuss.huggingface.co/u/Kamaljp)\ **Post date:** [June 26, 2023, 11:26am UTC](https://discuss.huggingface.co/t/problem-pushing-dataset-to-huggingface/26993/12 "2023-06-26T11:26:51Z")
The issue still continues even after using version 2.13. ---
**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\ **Post date:** [June 26, 2023, 2:01pm UTC](https://discuss.huggingface.co/t/problem-pushing-dataset-to-huggingface/26993/13 "2023-06-26T14:01:28Z")
If you still get the same error, restart the runtime for the version update to take effect.