# Using local dataset without changing cache

**URL:** <https://discuss.huggingface.co/t/using-local-dataset-without-changing-cache/53143>\
**Category:** 🤗Datasets\
**Created:** [September 1, 2023, 1:47pm UTC](https://discuss.huggingface.co/t/using-local-dataset-without-changing-cache/53143 "2023-09-01T13:47:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![cwallenwein](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cwallenwein/32/21752_2.png) [@cwallenwein](https://discuss.huggingface.co/u/cwallenwein)\
**Post date:** [September 1, 2023, 1:47pm UTC](https://discuss.huggingface.co/t/using-local-dataset-without-changing-cache/53143/1 "2023-09-01T13:47:27Z")

</div>

Hi friends👋  
I want to use RedPajama for training a language model. I have access to a folder where the entire dataset is already downloaded and now I want to load the local files with 🤗datasets for simple usage. Other people are relying on the downloaded data. Therefore I’m not allowed to change anything in that folder. How can I do that?

I’m currently using the following code to load the dataset

```auto
os.environ["RED_PAJAMA_DATA_DIR"] = PATH_TO_DOWNLOADED_RED_PAJAMA
ds = load_dataset("togethercomputer/RedPajama-Data-1T", "book")

```

But when I execute that code, it generates a training split and saves everything to the cache folder creating a file as large as the original dataset. I would like to use the downloaded data and limit the additional storage requirement of the code

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/9/1/91362c14797c55471d56ec9bb94e6143a6eda7bf.png)

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [September 6, 2023, 1:08pm UTC](https://discuss.huggingface.co/t/using-local-dataset-without-changing-cache/53143/2 "2023-09-06T13:08:59Z")

</div>

Hi! `load_dataset` saves a generated dataset in the Arrow format to be able to memory-map it later, so this is the expected behavior. If you are not interested in random access, you can pass `streaming=True` to get an `IterableDataset` that doesn’t require cache files.

---

<div class="post-metadata">

**Author:** ![cwallenwein](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cwallenwein/32/21752_2.png) [@cwallenwein](https://discuss.huggingface.co/u/cwallenwein)\
**Post date:** [September 6, 2023, 4:37pm UTC](https://discuss.huggingface.co/t/using-local-dataset-without-changing-cache/53143/3 "2023-09-06T16:37:37Z")

</div>

Perfect, thank you for this straightforward solution!