# \[solved\] How to load multiple arrow files into one dataset

**URL:** <https://discuss.huggingface.co/t/solved-how-to-load-multiple-arrow-files-into-one-dataset/49286>\
**Category:** Beginners\
**Created:** [August 3, 2023, 9:29am UTC](https://discuss.huggingface.co/t/solved-how-to-load-multiple-arrow-files-into-one-dataset/49286 "2023-08-03T09:29:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![aflip](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aflip/32/18198_2.png) [@aflip](https://discuss.huggingface.co/u/aflip)\
**Post date:** [August 3, 2023, 9:29am UTC](https://discuss.huggingface.co/t/solved-how-to-load-multiple-arrow-files-into-one-dataset/49286/1 "2023-08-03T09:29:58Z")

</div>

Hello

I am trying learn how to do embeddings etc. And in an earlier run, had saved some embeddings into arrow files using `dataset.save_to_disk( )` which generated two files `data-00000-of-00002.arrow and data-00001-of-00002.arrow`

Today, I want to load these two files into one dataset.

what Ive’ tried: `Dataset.from_file("/content/drive/MyDrive/data-00000-of-00002.arrow", "/content/drive/MyDrive/data-00001-of-00002.arrow")`, that is, passing the two files to `from_file`

but this gives me an error `AttributeError: 'str' object has no attribute 'copy'`

So how do i load these two into one dataset?

expected outcome: the orignial dataset that I had created and saved to disk is re-created from these two files.

thank you

Update: Unsure if this is a workaround or something but I used `from datasets import load_dataset` and then `data = load_dataset("arrow", data_files={file1, file2, file3})` instead of `Dataset.from_file` and this works. I am kinda glad my question went into the Akismet queue, because it made me want to try alternatives 🙂 If someone could tell me how to use the `Dataset.from_file` function do the same, I would love that.

Cheers

---

<div class="post-metadata">

**Author:** ![TurboPascal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/turbopascal/32/24246_2.png) [@TurboPascal](https://discuss.huggingface.co/u/TurboPascal)\
**Post date:** [August 25, 2023, 5:03am UTC](https://discuss.huggingface.co/t/solved-how-to-load-multiple-arrow-files-into-one-dataset/49286/2 "2023-08-25T05:03:35Z")

</div>

I have the same problem. Load from multiple arrow files into a dataset. Similar to the result after the map(). I want to process the arrow files I need in advance

---

<div class="post-metadata">

**Author:** ![aflip](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aflip/32/18198_2.png) [@aflip](https://discuss.huggingface.co/u/aflip)\
**Post date:** [August 25, 2023, 8:45am UTC](https://discuss.huggingface.co/t/solved-how-to-load-multiple-arrow-files-into-one-dataset/49286/3 "2023-08-25T08:45:07Z")

</div>

HI Enze, what do you mean process the files in advance? Are you able to load the files into a dataset? please add some details on what you’re trying to do

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [August 28, 2023, 6:22pm UTC](https://discuss.huggingface.co/t/solved-how-to-load-multiple-arrow-files-into-one-dataset/49286/4 "2023-08-28T18:22:44Z")

</div>

You can load multiple Arrow files using `Dataset.from_file` (assuming their schemas match, so no additional casts are required) as follows:

```python
from datasets import concatenate_datasets, Dataset

ds = concatenate_datasets([Dataset.from_file(arrow_file) for arrow_file in arrow_files])

```

---

<div class="post-metadata">

**Author:** ![TurboPascal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/turbopascal/32/24246_2.png) [@TurboPascal](https://discuss.huggingface.co/u/TurboPascal)\
**Post date:** [September 16, 2023, 7:01pm UTC](https://discuss.huggingface.co/t/solved-how-to-load-multiple-arrow-files-into-one-dataset/49286/5 "2023-09-16T19:01:02Z")

</div>

such as, to processing a dataset into multiple arrow files and then using load\_ from\_disk methods