# How to Use a Nested Python Dictionary in Dataset.from\_dict **URL:** \ **Category:** Beginners\ **Created:** [April 26, 2021, 12:21am UTC](https://discuss.huggingface.co/t/how-to-use-a-nested-python-dictionary-in-dataset-from-dict/5757 "2021-04-26T00:21:48Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![GSA](https://avatars.discourse-cdn.com/v4/letter/g/839c29/32.png) [@GSA](https://discuss.huggingface.co/u/GSA)\ **Post date:** [April 26, 2021, 12:21am UTC](https://discuss.huggingface.co/t/how-to-use-a-nested-python-dictionary-in-dataset-from-dict/5757/1 "2021-04-26T00:21:48Z")
I have a nested python dictionary # create a nested dictionary Dict = {‘train’: {‘id’: np.arange(len(train\_texts)), ‘tokens’: train\_texts, ‘tags’: train\_tags}, ‘val’: {‘id’: np.arange(len(val\_texts)), ‘tokens’: val\_texts, ‘tags’: val\_tags}, ‘test’: {‘id’: np.arange(len(test\_texts)), ‘tokens’: test\_texts, ‘tags’: test\_tags} } My question how do I use the nested dictionary in transformers Dataset.from\_dict() such that it gives me an output like the following: DatasetDict({ train: Dataset({ features: [‘id’, ‘tokens’, ‘tags’], num\_rows: 6801 }) val: Dataset({ features: [‘id’, ‘tokens’, ‘tags’], num\_rows: 1480 }) test: Dataset({ features: [‘id’, ‘tokens’, ‘tags’], num\_rows: 1532 }) }) ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [April 26, 2021, 10:59am UTC](https://discuss.huggingface.co/t/how-to-use-a-nested-python-dictionary-in-dataset-from-dict/5757/2 "2021-04-26T10:59:54Z")
hey @GSA, as far as i know you can’t create a `DatasetDict` object directly from a python `dict`, but you could try creating 3 `Dataset` objects (one for each split) and then add them to `DatasetDict` as follows: ```python dataset = DatasetDict() # using your `Dict` object for k,v in Dict.items(): dataset[k] = Dataset.from_dict(v) ``` ---
**Author:** ![GSA](https://avatars.discourse-cdn.com/v4/letter/g/839c29/32.png) [@GSA](https://discuss.huggingface.co/u/GSA)\ **Post date:** [April 26, 2021, 4:39pm UTC](https://discuss.huggingface.co/t/how-to-use-a-nested-python-dictionary-in-dataset-from-dict/5757/3 "2021-04-26T16:39:34Z")
@lewtun, Thanks for your help. It worked. Also, as a follow up question, how can I use the `datasets.ClassLabel` feature to specify a predefined set of classes which can have labels associated to them and be stored as integers in the dataset. This field will be stored and retrieved as an integer value and two conversion methods, datasets.`ClassLabel.str2int()` and datasets.`ClassLabel.int2str()` can be used to convert from the label names to the associate integer value and vice-versa. ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [April 26, 2021, 9:01pm UTC](https://discuss.huggingface.co/t/how-to-use-a-nested-python-dictionary-in-dataset-from-dict/5757/4 "2021-04-26T21:01:12Z")
great that it worked! > [@GSA](#): > > Also, as a follow up question, how can I use the `datasets.ClassLabel` feature to specify a predefined set of classes which can have labels associated to them and be stored as integers in the dataset. i’m not sure i understand the question (what exactly do you want to know?), but judging by your first comment it seems that you’re doing named entity recognition. you can see in the `datasets` library how `ClassLabel` is used for this task e.g. here [https://github.com/huggingface/datasets/blob/8e903b5ef7c039cee79f0a2da3dd328d59c38588/datasets/conll2003/conll2003.py#L167](https://github.com/huggingface/datasets/blob/8e903b5ef7c039cee79f0a2da3dd328d59c38588/datasets/conll2003/conll2003.py#L167) ---
**Author:** ![GSA](https://avatars.discourse-cdn.com/v4/letter/g/839c29/32.png) [@GSA](https://discuss.huggingface.co/u/GSA)\ **Post date:** [April 27, 2021, 2:52am UTC](https://discuss.huggingface.co/t/how-to-use-a-nested-python-dictionary-in-dataset-from-dict/5757/5 "2021-04-27T02:52:48Z")
@lewtun, Yes, you are correct, I’m trying to work on a named entity recognition task. So what I was hoping to achieve was to be able to do something like below: ```auto Dict = {'train': {'id': np.arange(len(train_texts)), 'tokens': train_texts, 'ner_tags': train_tags( datasets.features.ClassLabel( names=['B-ORG', 'I-ORG', 'O', 'I-EVENT', 'I-PERSON', 'B-PERSON'] ) ) }, 'val': {'id': np.arange(len(train_texts)), 'tokens': train_texts, 'ner_tags': train_tags( datasets.features.ClassLabel( names=['B-ORG', 'I-ORG', 'O', 'I-EVENT', 'I-PERSON', 'B-PERSON'] ) ) }, 'test': {'id': np.arange(len(train_texts)), 'tokens': train_texts, 'ner_tags': train_tags( datasets.features.ClassLabel( names=['B-ORG', 'I-ORG', 'O', 'I-EVENT', 'I-PERSON', 'B-PERSON'] ) ) } } ``` Specifically, to include the specification of the ClassLabel in the python dictionary statement. Doing it that way does not work because it gives a 'list object is not callable error`. Is there a way to work around that? ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [April 27, 2021, 12:51pm UTC](https://discuss.huggingface.co/t/how-to-use-a-nested-python-dictionary-in-dataset-from-dict/5757/6 "2021-04-27T12:51:43Z")
one way you can do this is by explicitly specifying the `features` argument in the `Dataset.from_dict` method ([docs](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=from_dict#datasets.Dataset.from_dict)), e.g. assume we have a `dict` with two examples: ```python from datasets import Dataset, ClassLabel, Sequence, Features, Value d = {'id': ['0', '1'], 'ner_tags': [[3, 0, 7, 0, 0, 0, 7, 0, 0], [1, 2]], 'tokens': [['EU', 'rejects', 'German', 'call', 'to', 'boycott', 'British', 'lamb', '.'], ['Peter', 'Blackburn']]} # define number of tags and their names tags = ClassLabel(num_classes=9, names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC']) # create dataset ds = Dataset.from_dict(mapping=d, features=Features({"ner_tags":Sequence(tags), 'id': Value(dtype='string'), 'tokens': Sequence(feature=Value(dtype='string'))})) # access ClassLabel feature - 0 returns tag "O", 1 returns "B-PER" etc ds.features["ner_tags"].feature.int2str(0) ``` you can consult the docs to see what role `Sequence` and `Value` play (tl;dr we have to explicitly define the datatypes for the underlying Arrow table) ---
**Author:** ![GSA](https://avatars.discourse-cdn.com/v4/letter/g/839c29/32.png) [@GSA](https://discuss.huggingface.co/u/GSA)\ **Post date:** [April 27, 2021, 10:04pm UTC](https://discuss.huggingface.co/t/how-to-use-a-nested-python-dictionary-in-dataset-from-dict/5757/7 "2021-04-27T22:04:08Z")
Thanks. That was extremely helpful.