# Downloading a dataset files locally

**URL:** <https://discuss.huggingface.co/t/downloading-a-dataset-files-locally/28309>\
**Category:** Beginners\
**Created:** [December 22, 2022, 7:28pm UTC](https://discuss.huggingface.co/t/downloading-a-dataset-files-locally/28309 "2022-12-22T19:28:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ampem](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@ampem](https://discuss.huggingface.co/u/ampem)\
**Post date:** [December 22, 2022, 7:28pm UTC](https://discuss.huggingface.co/t/downloading-a-dataset-files-locally/28309/1 "2022-12-22T19:28:00Z")

</div>

Due to proxies and various other restrictions and policies, I cannot download the data using the APIs like:

```auto
from datasets import load_dataset
raw_datasets = load_dataset("glue", "mrpc")

```

I had the same problem when downloading pretrain models, but there is an alternative, to download the model files and load the model locally, for example:

```auto
git lfs install
git clone https://huggingface.co/bert-base-uncased

```

Then i can use

```auto
model = AutoModelForSequenceClassification.from_pretrained("path/to/locally/downloaded/model/files")

```

Can I download the dataset files in a similar fashion directly and for example use? if yes how?

```auto
raw_datasets = load_dataset("path/to/locally/downloaded/dataset/files")

```

---

<div class="post-metadata">

**Author:** ![mapama247](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mapama247/32/33424_2.png) [@mapama247](https://discuss.huggingface.co/u/mapama247)\
**Post date:** [December 25, 2022, 10:12am UTC](https://discuss.huggingface.co/t/downloading-a-dataset-files-locally/28309/2 "2022-12-25T10:12:33Z")

</div>

You can use the `wget` command followed by the file’s URL, which should have the following format: `<HUB_REPO_URL>/resolve/main/<FILE_NAME>`. If you are unsure about the exact URL, you can just go to the “Files and versions” section and right-click the little arrow next to the file size to select the “Copy link address” option.

For instance, this would be a way to download the MRPC corpus that you mention:

> wget [https://huggingface.co/datasets/glue/resolve/main/dataset\_infos.json](https://huggingface.co/datasets/glue/resolve/main/dataset_infos.json)  
> wget [https://huggingface.co/datasets/glue/resolve/main/glue.py](https://huggingface.co/datasets/glue/resolve/main/glue.py)

And then you can enter python and do:

> from datasets import load\_dataset  
> mrpc = load\_dataset(“./glue.py”, “mrpc”)

---

<div class="post-metadata">

**Author:** ![mikechen](https://avatars.discourse-cdn.com/v4/letter/m/ecc23a/32.png) [@mikechen](https://discuss.huggingface.co/u/mikechen)\
**Post date:** [September 22, 2023, 4:23am UTC](https://discuss.huggingface.co/t/downloading-a-dataset-files-locally/28309/3 "2023-09-22T04:23:40Z")

</div>

Mapama247

It does not work.

First of all, it could not directly download the dataset . Second, even the above code does not work.

For instance, after downloading xsum.py, I use the following code and try to download the XSUM dataset.

```auto
from datasets import load_dataset
raw_datasets = load_dataset("./xsum.py", "raw_datasets", split="train)

```

It shows the error as follows.

_FileNotFoundError: Local file data/XSUM-EMNLP18-Summary-Data-Original.tar.gz doesn’t exist_

I find there is one line code in the xsum.py

```auto
# From https://github.com/EdinburghNLP/XSum/issues/12
_URL_DATA = "data/XSUM-EMNLP18-Summary-Data-Original.tar.gz"

```

**Option 1:**

It can download the dataset but ReadError while “Generating train split” if f I use the following code to replace the above “\_URL\_DATA…”

\_URL\_DATA = “[http://bollin.inf.ed.ac.uk/public/direct/XSUM-EMNLP18-Summary-Data-Original.tar.gz](http://bollin.inf.ed.ac.uk/public/direct/XSUM-EMNLP18-Summary-Data-Original.tar.gz)”

**Option 2**

After adding ssl code, it works if using the original code as follows.

```auto
import ssl

try:
    _create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
    pass
else:
    ssl._create_default_https_context = _create_unverified_https_context

# The xsum dataset is stored in .cache/huggingface/datasets/xsum
from datasets import load_dataset
raw_datasets = load_dataset("xsum", split="train")

```

Anyway, the method is not a direct method. It could not save the code locally.

The direct downloading method is listed as follows.

`$ wget http://bollin.inf.ed.ac.uk/public/direct/XSUM-EMNLP18-Summary-Data-Original.tar.gz --no-check-certificate`

However, it is not easy to get such a downloading weblink for every dataset in HuggingFace.

---

<div class="post-metadata">

**Author:** ![marccasalssalvador](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marccasalssalvador/32/33970_2.png) [@marccasalssalvador](https://discuss.huggingface.co/u/marccasalssalvador)\
**Post date:** [November 4, 2024, 11:30am UTC](https://discuss.huggingface.co/t/downloading-a-dataset-files-locally/28309/4 "2024-11-04T11:30:16Z")

</div>

Nice. Additionally, you can download using `git lfs`, and then using

```auto
git clone https://huggingface.co/datasets/user/dataset-name

```