# Doccano dataset for named entity recognition task using BERT

**URL:** <https://discuss.huggingface.co/t/doccano-dataset-for-named-entity-recognition-task-using-bert/85786>\
**Category:** Beginners\
**Created:** [May 8, 2024, 10:01am UTC](https://discuss.huggingface.co/t/doccano-dataset-for-named-entity-recognition-task-using-bert/85786 "2024-05-08T10:01:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ny-andry](https://avatars.discourse-cdn.com/v4/letter/n/3ab097/32.png) [@ny-andry](https://discuss.huggingface.co/u/ny-andry)\
**Post date:** [May 8, 2024, 10:01am UTC](https://discuss.huggingface.co/t/doccano-dataset-for-named-entity-recognition-task-using-bert/85786/1 "2024-05-08T10:01:46Z")

</div>

I’m trying to use BERT for custom token classification, I have annotated data for this task using doccano, the exported dataset in jsonl format looks like this:

```auto
{"text": "EU rejects German call to boycott British lamb.", "label": [[0, 2, "ORG"], [11, 17, "MIS"] ]}
{"text": "Peter Blackburn", "label": [[0, 15, "PER"] ]}
{"text": "President Obama", "label": [[10, 15, "PER"] ]}

```

Following [this](https://discuss.huggingface.co/t/problem-with-loading-custom-dataset-from-jsonl-file/38718), I got to load the data, but I can’t feed it to BERT yet.

I followed the tutorial on Huggingface for [token classification](https://huggingface.co/learn/nlp-course/en/chapter7/2), basically I was trying to get a dataset that look similar to this, just like the tutorial:

```auto
DatasetDict({
    train: Dataset({
        features: ['id', 'tokens', 'ner_tags'],
        num_rows: 14041
    })
    validation: Dataset({
        features: ['id', 'tokens', 'ner_tags'],
        num_rows: 3250
    })
    test: Dataset({
        features: ['id', 'tokens', 'ner_tags'],
        num_rows: 3453
    })
})

```

I’m trying to solve this by using pandas to tranform the data before using `Dataset.from_pandas`, but I can’t get it to work. Maybe there are better way to do this. Any help would be appreciated.

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [May 8, 2024, 11:51am UTC](https://discuss.huggingface.co/t/doccano-dataset-for-named-entity-recognition-task-using-bert/85786/2 "2024-05-08T11:51:04Z")

</div>

Hi,

That’s already great! Citing myself here 🙂 [NER model fine tuning with labeled spans - #3 by nielsr](https://discuss.huggingface.co/t/ner-model-fine-tuning-with-labeled-spans/10633/3)

---

<div class="post-metadata">

**Author:** ![ny-andry](https://avatars.discourse-cdn.com/v4/letter/n/3ab097/32.png) [@ny-andry](https://discuss.huggingface.co/u/ny-andry)\
**Post date:** [May 14, 2024, 5:55pm UTC](https://discuss.huggingface.co/t/doccano-dataset-for-named-entity-recognition-task-using-bert/85786/3 "2024-05-14T17:55:59Z")

</div>

I found the solution, using this [doccano/doccano-transformer](https://github.com/doccano/doccano-transformer) from the docanno team. There is a catch though, it has an annoying bug solved by [this](https://github.com/doccano/doccano-transformer/issues/35), and since there is next to no documentation, we can use [this](https://github.com/doccano/doccano-transformer/issues/28) to write the file. 👌

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [May 15, 2024, 5:56am UTC](https://discuss.huggingface.co/t/doccano-dataset-for-named-entity-recognition-task-using-bert/85786/4 "2024-05-15T05:56:11Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.