# Creating a dataset with custom data

**URL:** <https://discuss.huggingface.co/t/creating-a-dataset-with-custom-data/22462>\
**Category:** Beginners\
**Created:** [September 2, 2022, 4:26pm UTC](https://discuss.huggingface.co/t/creating-a-dataset-with-custom-data/22462 "2022-09-02T16:26:32Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![edensn](https://avatars.discourse-cdn.com/v4/letter/e/b5e925/32.png) [@edensn](https://discuss.huggingface.co/u/edensn)\
**Post date:** [September 2, 2022, 4:26pm UTC](https://discuss.huggingface.co/t/creating-a-dataset-with-custom-data/22462/1 "2022-09-02T16:26:32Z")

</div>

Hey there, I’m trying to create a DatasetDict with two datasets(train and dev) for fine tuning a bart model.

I’ve created lists of source sentences, target sentences and id’s, they are lists of strings.

```auto
data = DatasetDict({
    "train": Dataset.from_dict({
        "id": train_idxs,
        "translation": {
            "source": train_inputs,
            "target": train_labels
        }
    }, features=Features({"id": Value(dtype='string'), "translation": {"source": Sequence, "target": Sequence}})),
    "dev": {
        "id": dev_idxs,
        "translation": {
            "source": dev_inputs,
            "target": dev_labels
        }
    }
})

```

is the code I’m using to create the DatasetDict, but I get error

```auto
TypeError: string indices must be integers

```

I want the object to have the same structure as the “Books” datasetdict that is used in this guide [Translation](https://huggingface.co/docs/transformers/tasks/translation)

if anyone has any suggestions please let me know, as well as if I need to provide more information!

Thank you!

---

<div class="post-metadata">

**Author:** ![NimaBoscarino](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nimaboscarino/32/4358_2.png) [@NimaBoscarino](https://discuss.huggingface.co/u/NimaBoscarino)\
**Post date:** [September 2, 2022, 11:48pm UTC](https://discuss.huggingface.co/t/creating-a-dataset-with-custom-data/22462/2 "2022-09-02T23:48:55Z")

</div>

Hello! I don’t think that you should need to manually invoke `DatasetDict`. Instead, since you’re loading text data you can use one of the methods listed here: [Load](https://huggingface.co/docs/datasets/loading#csv)

If you have a bunch of CSV fileswith the headers `source` and `target`, you can load them with

```python
from datasets import load_dataset
data = load_dataset("csv", data_files={"train": ["my_text_1.txt", "my_text_2.txt"], "validate": "my_test_3.txt"})

```

In general, I think it’s best to avoid `DatasetDict` and instead rely on the helper methods as much as possible.

Hope that helps!

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [September 5, 2022, 10:51am UTC](https://discuss.huggingface.co/t/creating-a-dataset-with-custom-data/22462/3 "2022-09-05T10:51:00Z")

</div>

Hi @edensn! You can use the following features dictionary instead of the current one to match the features:

```python
Features({"id": Value(dtype='string'), "translation": Translation(languages=["source", "target"])})

```

---

<div class="post-metadata">

**Author:** ![edensn](https://avatars.discourse-cdn.com/v4/letter/e/b5e925/32.png) [@edensn](https://discuss.huggingface.co/u/edensn)\
**Post date:** [September 5, 2022, 1:57pm UTC](https://discuss.huggingface.co/t/creating-a-dataset-with-custom-data/22462/4 "2022-09-05T13:57:33Z")

</div>

Thank you both of you guys for the suggestions! I actually figured it out using the method that @mariosasko suggested by just opening the Books dataset I was trying to mimic and doing just looking at books.features!