# Token alignment for word-level tasks

**URL:** <https://discuss.huggingface.co/t/token-alignment-for-word-level-tasks/577>\
**Category:** 🤗Tokenizers\
**Created:** [August 3, 2020, 4:46pm UTC](https://discuss.huggingface.co/t/token-alignment-for-word-level-tasks/577 "2020-08-03T16:46:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [August 3, 2020, 4:46pm UTC](https://discuss.huggingface.co/t/token-alignment-for-word-level-tasks/577/1 "2020-08-03T16:46:31Z")

</div>

Hey guys,

I want to make a POS tagger on top of BERT. The dataset is in conll-u format [1], input sentences are already tokenized; input word tokens are mapped to labels (POS etc ). Therefore, I have to take special care of input/output alignment as BERT will add additional tokens during tokenization, similar to what is described in the original BERT repo [2].

So I took the alignment algorithm outlined in [2]; I tokenized each input word and map/expanded labels for each BERT tokenized word. I then needed to add padding to each sentence, attach attention\_mask, convert to tensors etc. Looking at Tokenizer API I didn’t see how to do this easily if I already have sentences and labels converted to token ids in the first step.

This whole exercise left me wondering if there is a simpler and less verbose approach to these word-level tasks using Tokenizer API?

Cheers,  
Vladimir

[1] [https://universaldependencies.org/format#conll-u-format](https://universaldependencies.org/format#conll-u-format)  
[2] [https://github.com/google-research/bert#tokenization](https://github.com/google-research/bert#tokenization)

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [August 5, 2020, 10:07am UTC](https://discuss.huggingface.co/t/token-alignment-for-word-level-tasks/577/2 "2020-08-05T10:07:10Z")

</div>

I found an answer in [utils\_ner.py](https://github.com/huggingface/transformers/blob/master/examples/token-classification/utils_ner.py) See function convert\_examples\_to\_features; it does what I was doing except in a more general model-agnostic approach.

Cheers,  
Vladimir