# All my sequences get tokenized the same

**URL:** <https://discuss.huggingface.co/t/all-my-sequences-get-tokenized-the-same/14657>\
**Category:** 🤗Tokenizers\
**Created:** [February 12, 2022, 5:16pm UTC](https://discuss.huggingface.co/t/all-my-sequences-get-tokenized-the-same/14657 "2022-02-12T17:16:53Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![timaie](https://avatars.discourse-cdn.com/v4/letter/t/dbc845/32.png) [@timaie](https://discuss.huggingface.co/u/timaie)\
**Post date:** [February 12, 2022, 5:16pm UTC](https://discuss.huggingface.co/t/all-my-sequences-get-tokenized-the-same/14657/1 "2022-02-12T17:16:53Z")

</div>

I’m using ProtBert models ([GitHub - agemagician/ProtTrans: ProtTrans is providing state of the art pretrained language models for proteins. ProtTrans was trained on thousands of GPUs from Summit and hundreds of Google TPUs using Transformers Models.](https://github.com/agemagician/ProtTrans)) for protein embeddings. When using the BertTokenizer (and similarly for T5Tokenizer), all my sequences get tokenized to [2, 1, 3]. Does anyone know what’s going wrong?

```auto
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("Rostlab/prot_bert", do_lower_case=False)
sequences = ['MEGGGKPNSASNSRDDGNSVYPSKAPATGPAAAD', 'MQLKAKEELLRNMELGLIPDQEIRQLIRVE', 'MTVSTSKTPKKNIKYTLTHTLQKWKETLKKITHETLSSI']

tokenizer(sequences)

```

gves a result like

```auto
{'input_ids': [[2, 1, 3], [2, 1, 3], [2, 1, 3]], 'token_type_ids': [[0, 0, 0], [0, 0, 0], [0, 0, 0]], 'attention_mask': [[1, 1, 1], [1, 1, 1], [1, 1, 1]]}

```

---

<div class="post-metadata">

**Author:** ![beneyal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/beneyal/32/27988_2.png) [@beneyal](https://discuss.huggingface.co/u/beneyal)\
**Post date:** [February 12, 2022, 5:22pm UTC](https://discuss.huggingface.co/t/all-my-sequences-get-tokenized-the-same/14657/2 "2022-02-12T17:22:31Z")

</div>

Hi there!

You need to use spaces between the string characters. `[2, 1, 3]` is `[CLS] [UNK] [SEP]`.

With spaces, it looks better 🙂

```python
>>> tokenizer(' '.join(list("MEGGGKPNSASNSRDDGNSVYPSKAPATGPAAAD")))

{'input_ids': [2, 21, 9, 7, 7, 7, 12, 16, 17, 10, 6, 10, 17, 10, 13, 14, 14, 7, 17, 10, 8, 20, 16, 10, 12, 6, 16, 6, 15, 7, 16, 6, 6, 6, 14, 3],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

```

---

<div class="post-metadata">

**Author:** ![timaie](https://avatars.discourse-cdn.com/v4/letter/t/dbc845/32.png) [@timaie](https://discuss.huggingface.co/u/timaie)\
**Post date:** [February 12, 2022, 5:24pm UTC](https://discuss.huggingface.co/t/all-my-sequences-get-tokenized-the-same/14657/3 "2022-02-12T17:24:24Z")

</div>

I just also realized that, thanks anyway! 🙂