# Data Conversion to Conll2003

**URL:** <https://discuss.huggingface.co/t/data-conversion-to-conll2003/56879>\
**Category:** Intermediate\
**Created:** [September 30, 2023, 3:33am UTC](https://discuss.huggingface.co/t/data-conversion-to-conll2003/56879 "2023-09-30T03:33:03Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sushantsur23](https://avatars.discourse-cdn.com/v4/letter/s/53a042/32.png) [@sushantsur23](https://discuss.huggingface.co/u/sushantsur23)\
**Post date:** [September 30, 2023, 3:33am UTC](https://discuss.huggingface.co/t/data-conversion-to-conll2003/56879/1 "2023-09-30T03:33:03Z")

</div>

I have downloaded a annotation.json file from the spacy free annotator NER tool. I am trying to convert the dataset in the format of conll2003. With the below code I am able to convert to the structure but I;m not sure how to add the features for the NER Tags in numbers. like if we run the below code conll2003[“train”].features[“ner\_tags”] I see the output as per below. I’m unable to do the same to my dataset additionally, kindly let me how to create token in the same way as per the hugging face dataset conll2003.  
Sequence(feature=ClassLabel(names=[‘O’, ‘B-PER’, ‘I-PER’, ‘B-ORG’, ‘I-ORG’, ‘B-LOC’, ‘I-LOC’, ‘B-MISC’, ‘I-MISC’], id=None), length=-1, id=None)

#Code to create the dataset structure  
from sklearn.model\_selection import train\_test\_split  
from datasets import Dataset, DatasetDict  
train\_data, test\_data = train\_test\_split(df, test\_size=0.2, random\_state=42)

# Splitting the training data into training and validation sets

train\_data, validation\_data = train\_test\_split(train\_data, test\_size=0.1, random\_state=42)

train = Dataset.from\_pandas(train\_data)  
test = Dataset.from\_pandas(test\_data)  
val = Dataset.from\_pandas(validation\_data)

conll2003 = DatasetDict()  
conll2003[‘train’] = train  
conll2003[‘validation’] = val  
conll2003[‘test’] = test

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [October 3, 2023, 2:18pm UTC](https://discuss.huggingface.co/t/data-conversion-to-conll2003/56879/2 "2023-10-03T14:18:10Z")

</div>

Hi! You can run `conll2003 = conll2003.cast_column("ner_tags", Sequence(ClassLabel(names=list_of_labels)))` to cast the NER tags to `Sequence(ClassLabel(...))`

---

<div class="post-metadata">

**Author:** ![sushantsur23](https://avatars.discourse-cdn.com/v4/letter/s/53a042/32.png) [@sushantsur23](https://discuss.huggingface.co/u/sushantsur23)\
**Post date:** [October 19, 2023, 3:50pm UTC](https://discuss.huggingface.co/t/data-conversion-to-conll2003/56879/3 "2023-10-19T15:50:48Z")

</div>

Hi Mario, Thank you so much for the response. I updated the code as per below but I’m getting error AttributeError: ‘DataFrame’ object has no attribute ‘cast\_column’ . I’m unable to go ahead on this. Is it possible to have a quick connect as per your convenience? it would be a big time help.

from datasets import ClassLabel, Sequence

list\_of\_labels = [‘O’, ‘DEBIT\_AMT’, ‘CREDIT\_AMT’, ‘BALANCE\_AMT’, ‘BANK\_NAME’, ‘ACCOUNT\_NO’, ‘CARD\_NO’]

df = df.cast\_column(“ner\_tags”, Sequence(ClassLabel(names=list\_of\_labels)))

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [October 20, 2023, 2:19pm UTC](https://discuss.huggingface.co/t/data-conversion-to-conll2003/56879/4 "2023-10-20T14:19:25Z")

</div>

`cast_column` should be called on a `Dataset`/`DatasetDict` object, not Pandas `DataFrame`

---

<div class="post-metadata">

**Author:** ![sushantsur23](https://avatars.discourse-cdn.com/v4/letter/s/53a042/32.png) [@sushantsur23](https://discuss.huggingface.co/u/sushantsur23)\
**Post date:** [December 28, 2023, 7:26am UTC](https://discuss.huggingface.co/t/data-conversion-to-conll2003/56879/5 "2023-12-28T07:26:09Z")

</div>

Thank you it worked, Any idea how can we create the pos\_tags and chunk\_tags?