# How to load dataset locally?

**URL:** <https://discuss.huggingface.co/t/how-to-load-dataset-locally/11601>\
**Category:** 🤗Datasets\
**Created:** [November 11, 2021, 3:11pm UTC](https://discuss.huggingface.co/t/how-to-load-dataset-locally/11601 "2021-11-11T15:11:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![fyyfu](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@fyyfu](https://discuss.huggingface.co/u/fyyfu)\
**Post date:** [November 11, 2021, 3:11pm UTC](https://discuss.huggingface.co/t/how-to-load-dataset-locally/11601/1 "2021-11-11T15:11:00Z")

</div>

I want to load dataset locally. (such as xcopa).

for xcopa, i manually download the datasets from this Link, and set the mode to offline mode. The code is:

```auto
import os
os.environ['HF_DATASETS_OFFLINE'] ='1'
from datasets import load_dataset
xcopa = load_dataset('./datasets/datasets/xcopa/xcopa.py', 
                     name='et',
                     date_dir='path/to/zip', 
                     cache_dir='path/to/cache')

```

But it still want to download the zip files from the Link rather than the locally files.  
So i compare the difference between the xcopa.py & matinf.py (which should manually download). They do have some differences, so Can xcopa dataset be load locally (manually download the zip files and load it)? (Maybe in the `_split_generators` method of the xcopa.py)

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [November 11, 2021, 4:52pm UTC](https://discuss.huggingface.co/t/how-to-load-dataset-locally/11601/2 "2021-11-11T16:52:47Z")

</div>

Hi,

I’ll assume you are trying to download the dataset that way due to an error thrown by `load_dataset("xcopa", "et")`. It’s going to be fixed soon!

As you correctly pointed out, there are some differences in the data that are causing the error. In the meantime, you can bypass the error and download the dataset (with the newest data) as follows:

```python
from datasets import load_dataset
dset = load_dataset("xcopa", "et", ignore_verifications=True)

```

---

<div class="post-metadata">

**Author:** ![fyyfu](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@fyyfu](https://discuss.huggingface.co/u/fyyfu)\
**Post date:** [November 12, 2021, 2:20am UTC](https://discuss.huggingface.co/t/how-to-load-dataset-locally/11601/3 "2021-11-12T02:20:50Z")

</div>

Thanks for your reply! But it seems cannot slove the problems. I want to manually download the original xcopa zip files to the local storage, And load this original files locally.

To some reasons, i cannot access the internet connection. So i use the above `path='./datasets/datasets/xcopa/xcopa.py'` .  
But in the xcopa.py, it seems must to download the zip files remotely.(Which might be different from the matinf.py? It provide the `dl_manager.manual_dir` in the Line 134. ).

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [November 12, 2021, 10:53am UTC](https://discuss.huggingface.co/t/how-to-load-dataset-locally/11601/4 "2021-11-12T10:53:52Z")

</div>

Hi,

oh, I see. The `_split_generators` method is the right place to use `dl_manager.manual_dir`. You can find an example script that uses it [here](https://github.com/huggingface/datasets/blob/master/datasets/ubuntu_dialogs_corpus/ubuntu_dialogs_corpus.py#L101). Additionally, you can use `dl_manager.extract(manual_dir)` to extract the data, so you don’t have to do it manually or with the `zipfile `module. And to avoid the download, comment out the line with the `dl_manager.download_and_extract(_URL)` call.

---

<div class="post-metadata">

**Author:** ![fyyfu](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@fyyfu](https://discuss.huggingface.co/u/fyyfu)\
**Post date:** [November 12, 2021, 12:28pm UTC](https://discuss.huggingface.co/t/how-to-load-dataset-locally/11601/5 "2021-11-12T12:28:44Z")

</div>

ok. Thanks a lot !  
That is to say, under the `datasets/datasets/xcopa/xcopa.py` in the `datasets` repository. I must to download the xcopa zip files remotely and get the cached files. Then i can load the cached datasets locally.

If i want to manully download the xcopa zip files and load it, i must to write another ‘xcopa.py’ (like you provided) to load the zip files locally. right? 👀