# Convert dataframe to NER dataset format

**URL:** <https://discuss.huggingface.co/t/convert-dataframe-to-ner-dataset-format/17813>\
**Category:** 🤗Datasets\
**Created:** [May 11, 2022, 10:17am UTC](https://discuss.huggingface.co/t/convert-dataframe-to-ner-dataset-format/17813 "2022-05-11T10:17:40Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![RuudVelo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ruudvelo/32/20334_2.png) [@RuudVelo](https://discuss.huggingface.co/u/RuudVelo)\
**Post date:** [May 11, 2022, 10:17am UTC](https://discuss.huggingface.co/t/convert-dataframe-to-ner-dataset-format/17813/1 "2022-05-11T10:17:40Z")

</div>

Hi. I am trying to convert a dataframe to the format for NER I have seen in example notebook. See below and example of the format  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/1baecfcfef81dee674d38c35a93f45b33f369caf.png)

I have a dataframe which looks like:  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/6/6180aee7974f2a176790f187d5b5af2a3ce52984.png)

The ner\_tags is an object column

If I convert this dataframe to a datasets format by using Dataset.from\_pandas(data) I get:  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/3/3fdfa208f369c1e0899e33c38fd7c3bd7381c038.png)

As you can see the format of the ‘ner\_tags’ is not the same as in the NER dataset example (tokens are good). I am struggling to get this in the same format for the ner\_tags. Can someone make a suggestion on this?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [May 16, 2022, 2:01pm UTC](https://discuss.huggingface.co/t/convert-dataframe-to-ner-dataset-format/17813/2 "2022-05-16T14:01:38Z")

</div>

Hi ! It looks like your the `ner_tags` column in your dataframe contains data of type list of one string, instead of list of integers. I guess you are loading your dataset from a CSV file ? When you have nested data, I would recommend using JSON Lines instead of CSV.

Anyway, to fix is you can process the `ner_tags` column to get lists of integers this way 🙂

```python
dataset = dataset.map(lambda x: {"ner_tags": [int(i) for i in x["ner_tags"].split(",")]})

```