# Writing load\_dataset loading script with multiple code files

**URL:** <https://discuss.huggingface.co/t/writing-load-dataset-loading-script-with-multiple-code-files/39539>\
**Category:** 🤗Datasets\
**Created:** [May 11, 2023, 1:11pm UTC](https://discuss.huggingface.co/t/writing-load-dataset-loading-script-with-multiple-code-files/39539 "2023-05-11T13:11:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![borgr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/borgr/32/24353_2.png) [@borgr](https://discuss.huggingface.co/u/borgr)\
**Post date:** [May 11, 2023, 1:11pm UTC](https://discuss.huggingface.co/t/writing-load-dataset-loading-script-with-multiple-code-files/39539/1 "2023-05-11T13:11:13Z")

</div>

How can I break the loading script into multiple code files (all in the dataset repo)? (e.g. add a loader in one file and a preprocessing format class from another)

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [May 12, 2023, 9:33am UTC](https://discuss.huggingface.co/t/writing-load-dataset-loading-script-with-multiple-code-files/39539/2 "2023-05-12T09:33:43Z")

</div>

You can do relative imports 🙂

e.g. if you have `my_dataset.py` containing the loader and `processing.py` containing your processing, you can import the processing into the loader:

```python
from .processing import my_process_fn

```

---

<div class="post-metadata">

**Author:** ![Elron](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/elron/32/5352_2.png) [@Elron](https://discuss.huggingface.co/u/Elron)\
**Post date:** [May 18, 2023, 3:53pm UTC](https://discuss.huggingface.co/t/writing-load-dataset-loading-script-with-multiple-code-files/39539/3 "2023-05-18T15:53:57Z")

</div>

What about a folder with json files in the dataset repo? is there any way to access it locally?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [May 25, 2023, 3:18pm UTC](https://discuss.huggingface.co/t/writing-load-dataset-loading-script-with-multiple-code-files/39539/4 "2023-05-25T15:18:32Z")

</div>

For data files like JSON you must use the `dl_manager.download()` method in the dataset script.

This way when loaded from the cloud it will download the data files, and locally it will simply return the path to the local file (nothing to download)