# Loading Dataset from Cache Data

**URL:** <https://discuss.huggingface.co/t/loading-dataset-from-cache-data/109517>\
**Category:** Intermediate\
**Created:** [September 30, 2024, 6:43pm UTC](https://discuss.huggingface.co/t/loading-dataset-from-cache-data/109517 "2024-09-30T18:43:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![abirmhadi](https://avatars.discourse-cdn.com/v4/letter/a/54ee81/32.png) [@abirmhadi](https://discuss.huggingface.co/u/abirmhadi)\
**Post date:** [September 30, 2024, 6:43pm UTC](https://discuss.huggingface.co/t/loading-dataset-from-cache-data/109517/1 "2024-09-30T18:43:14Z")

</div>

!!SOLVED!!

Indeed I had a corrupt cache file. Just re-downloaded that specific file solved the issue.

N.B.: The environment variables also need to be set before importing anything related to datasets lib.

* * *

Hi there!

I’m trying to load pixparse/cc3m-wds dataset from cached data. Long story short, my cluster network is having some issues reaching hugging face hub and I cannot directly download dataset using the library. So I used Google Colab to download the dataset into a cache folder. I have since downloaded the entire cache folder to my node disk. Now trying to load dataset and build from the existing downloaded shards from this cache folder using,

```auto
ds = load_dataset("pixparse/cc3m-wds", cache_dir="./cc3m/")

```

But it keeps spitting out:

```auto
File ~/.conda/envs/lavis/lib/python3.8/site-packages/datasets/table.py:1017, in MemoryMappedTable.from_file(cls, filename, replays)
   1015 @classmethod
   1016 def from_file(cls, filename: str, replays=None):
-> 1017 table = _memory_mapped_arrow_table_from_file(filename)
   1018 table = cls._apply_replays(table, replays)
   1019 return cls(table, filename, replays)

File ~/.conda/envs/lavis/lib/python3.8/site-packages/datasets/table.py:64, in _memory_mapped_arrow_table_from_file(filename)
     62 def _memory_mapped_arrow_table_from_file(filename: str) -> pa.Table:
     63 opened_stream = _memory_mapped_record_batch_reader_from_file(filename)
---> 64 pa_table = opened_stream.read_all()
     65 return pa_table

File ~/.conda/envs/lavis/lib/python3.8/site-packages/pyarrow/ipc.pxi:757, in pyarrow.lib.RecordBatchReader.read_all()

File ~/.conda/envs/lavis/lib/python3.8/site-packages/pyarrow/error.pxi:91, in pyarrow.lib.check_status()

OSError: Expected to be able to read 8457224 bytes for message body, got 6054384

```

I double checked the downloaded files and matched their size. In google colab it progresses to,

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/b/2/b2154d0c5ae57166bfa0f838b333dc65319f8dfb.png)

But runs out of disk memory as expected. Can someone help me rebuild this dataset from the cache I have in my HPC node?

Thanks!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [September 30, 2024, 10:56pm UTC](https://discuss.huggingface.co/t/loading-dataset-from-cache-data/109517/2 "2024-09-30T22:56:20Z")

</div>

The save method is correct, but the load method may be wrong.  
The cache\_dir= specified at load time is the **location of HF’s proprietary cache** , and it will not work properly if the file or configuration file is not located in the correct place, even slightly.  
Why don’t you forget about cache\_dir= and use **path=** to directly specify the path where the dataset entities are located?  
This basically works as long as you have the dataset file.

> **[Loading methods](https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.load_dataset)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.