# Load dataset from a specific cache file

**URL:** <https://discuss.huggingface.co/t/load-dataset-from-a-specific-cache-file/24994>\
**Category:** 🤗Datasets\
**Created:** [October 26, 2022, 8:13am UTC](https://discuss.huggingface.co/t/load-dataset-from-a-specific-cache-file/24994 "2022-10-26T08:13:29Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mdelas](https://avatars.discourse-cdn.com/v4/letter/m/b5ac83/32.png) [@mdelas](https://discuss.huggingface.co/u/mdelas)\
**Post date:** [October 26, 2022, 8:13am UTC](https://discuss.huggingface.co/t/load-dataset-from-a-specific-cache-file/24994/1 "2022-10-26T08:13:29Z")

</div>

Dear community,

Does anyone know if there is a way to load dataset from a specific cached file? I applied very costly transformations using a map function to do some data augmentation and finally, I had my dataset. However, now in my main script, I would like to use the cached file built from another script. Is there any way to call a specific cache file from `~/.cache/huggingface/datasets/` ?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [October 26, 2022, 11:12am UTC](https://discuss.huggingface.co/t/load-dataset-from-a-specific-cache-file/24994/2 "2022-10-26T11:12:02Z")

</div>

You can reload any arrow file from the cache with

```python
from datasets import Dataset 

ds = Dataset.from_file("path/to/data.arrow")

```

---

<div class="post-metadata">

**Author:** ![jahb57](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jahb57/32/20674_2.png) [@jahb57](https://discuss.huggingface.co/u/jahb57)\
**Post date:** [February 26, 2024, 3:48pm UTC](https://discuss.huggingface.co/t/load-dataset-from-a-specific-cache-file/24994/3 "2024-02-26T15:48:13Z")

</div>

Does `from_file` try to read the whole dataset?

I am in a similar place where I spent days adding columns to a dataset using `.map()` but the execution failed when I tried to `save_to_disk`. So I am hoping to save it from the cache.  
The map operation ended successfully making the dataset huge (1.3TB)

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/d/b/db8910622dd1cac058458a4997b3e592ea6202c9.png)

Originally i planned to load the dataset as an iterable to only use enough rows in a batch that will fit in Memory, I am currently trying Datasets.from\_file(“the larger arrow file”). But it’s really taking its time so I am wondering if this is still usable. I am also not sure which arrow file I should be trying to read I am hoping that the one with the `cache`-prefix contains all the data and not just part of it.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [February 26, 2024, 6:15pm UTC](https://discuss.huggingface.co/t/load-dataset-from-a-specific-cache-file/24994/4 "2024-02-26T18:15:53Z")

</div>

The “cache-9aaxxxxx” file should be the one indeed 🙂

`Dataset.from_file` should work - what takes time is reading the metadata of all the record batches (=chunks of arrow files). It doesn’t load the actual dataset content in memory.

Alternatively you can use `IterableDataset.from_file` which doesn’t read the metadata, but we haven’t implemented `save_to_disk` for `IterableDataset`