# Loading train and test splits with \`audiofolder\`

**URL:** <https://discuss.huggingface.co/t/loading-train-and-test-splits-with-audiofolder/22447>\
**Category:** 🤗Datasets\
**Created:** [September 2, 2022, 9:49am UTC](https://discuss.huggingface.co/t/loading-train-and-test-splits-with-audiofolder/22447 "2022-09-02T09:49:49Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [September 2, 2022, 9:49am UTC](https://discuss.huggingface.co/t/loading-train-and-test-splits-with-audiofolder/22447/1 "2022-09-02T09:49:49Z")

</div>

Hi folks,

I’m using the new `audiofolder` feature ([docs](https://huggingface.co/docs/datasets/main/en/audio_load#audiofolder)) to load audio files from this [Kaggle dataset](https://www.kaggle.com/competitions/kaggle-pog-series-s01e02/overview). My folder structure has the following form:

```auto
.
├── test
| ├── 01.ogg
| ├── 02.ogg
| ├── ...
│ └── metadata.csv
└── train
    ├── 01.ogg
    ├── 02.ogg
    ├── ...
    └── metadata.csv

```

In particular, the train and test splits have _different_ features. Now, when I try to load both splits at once with:

```python
from datasets import load_dataset

ds = load_dataset("audiofolder", data_dir="path/to/dir")

```

I get the following error:

```auto
ValueError: Metadata files /Users/lewtun/Downloads/kaggle-pog-series-s01e02/test/metadata.csv and /Users/lewtun/Downloads/kaggle-pog-series-s01e02/train/metadata.csv have different features: ('/Users/lewtun/Downloads/kaggle-pog-series-s01e02/train/metadata.csv', {'song_id': Value(dtype='int64', id=None), 'file_name': Value(dtype='string', id=None), 'filepath': Value(dtype='string', id=None), 'genre_id': Value(dtype='int64', id=None), 'genre': Value(dtype='string', id=None)}) != {'song_id': Value(dtype='int64', id=None), 'file_name': Value(dtype='string', id=None), 'filepath': Value(dtype='string', id=None)}

```

This suggests that `audiofolder` is trying to combine the train and test splits as a _single dataset_ - is this correct?

I am able to load each split separately, but my question is: can `audiofolder` load multiple splits as a `DatasetDict` object?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [September 2, 2022, 9:56am UTC](https://discuss.huggingface.co/t/loading-train-and-test-splits-with-audiofolder/22447/2 "2022-09-02T09:56:47Z")

</div>

For `ImageFolder`, it works as follows:

```auto
dataset = load_dataset("imagefolder", data_files={"train": ["path/to/file1", "path/to/file2"], "test": ["path/to/file3", "path/to/file4"]})

```

So I assume it’s similar for `AudioFolder`.

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [September 2, 2022, 10:09am UTC](https://discuss.huggingface.co/t/loading-train-and-test-splits-with-audiofolder/22447/3 "2022-09-02T10:09:15Z")

</div>

Thanks @nielsr !

Your suggestion works provided I don’t include `metadata.csv` in the `data_files` dictionary, e.g. the following works:

```auto
ds = load_dataset("audiofolder", data_files={"train": ["train/01.ogg"], "test": ["test/01.ogg"]})

```

but the following fails (with the same error as above):

```auto
ds = load_dataset("audiofolder", data_files={"train": ["train/metadata.csv", "train/01.ogg"], "test": ["test/metadata.csv", "test/01.ogg"]})

```

I think the key issue here is that I want to load the audio files with _metadata_ and `audiofolder` seems to only support loading a single folder at a time.

---

<div class="post-metadata">

**Author:** ![polinaeterna](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/polinaeterna/32/19055_2.png) [@polinaeterna](https://discuss.huggingface.co/u/polinaeterna)\
**Post date:** [September 2, 2022, 7:11pm UTC](https://discuss.huggingface.co/t/loading-train-and-test-splits-with-audiofolder/22447/4 "2022-09-02T19:11:46Z")

</div>

hi @lewtun !  
yes, `audiofolder` can load multiple splits as a `DatasetDict` object 🙂 they will be inferred from the directory structure or file names.  
The problem here is that if more then one metadata files are found and they contain different set of features, you would have an error with both `data_dir` and `data_files` approach - this is an intentional behavior. In this case, you can check that without metadata it would work by setting `drop_metadata=True` - you will have two splits in a `DatasetDict` object.

for now the only option to load two splits with different set of features with Audio/ImageFolder is to load them as separate datasets:

```auto
ds_train = load_dataset("audiofolder", data_dir="path/to/dir/train")
ds_test = load_dataset("audiofolder", data_files={"test": "path/to/dir/test/**"})

```

\*you need to use `data_files` if you want to give a specific name to a split. By default if no split structure is found, all the data goes to the “train” split. The relevant code is here: [datasets/data\_files.py at main · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/blob/main/src/datasets/data_files.py#L27)

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [September 5, 2022, 8:57am UTC](https://discuss.huggingface.co/t/loading-train-and-test-splits-with-audiofolder/22447/5 "2022-09-05T08:57:20Z")

</div>

All the splits of a dataset must have the same features.

In your test set you can add the missing features and set them to None if you want

---

<div class="post-metadata">

**Author:** ![doof-ferb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/doof-ferb/32/23404_2.png) [@doof-ferb](https://discuss.huggingface.co/u/doof-ferb)\
**Post date:** [February 10, 2024, 11:11am UTC](https://discuss.huggingface.co/t/loading-train-and-test-splits-with-audiofolder/22447/6 "2024-02-10T11:11:19Z")

</div>

my workaround is to merge 2 `metadata.csv` into only 1 and put at root