# Tokenization compared to sentencepiece

**URL:** <https://discuss.huggingface.co/t/tokenization-compared-to-sentencepiece/106278>\
**Category:** 🤗Tokenizers\
**Created:** [September 11, 2024, 8:43am UTC](https://discuss.huggingface.co/t/tokenization-compared-to-sentencepiece/106278 "2024-09-11T08:43:27Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![marcin-ochman](https://avatars.discourse-cdn.com/v4/letter/m/f4b2a3/32.png) [@marcin-ochman](https://discuss.huggingface.co/u/marcin-ochman)\
**Post date:** [September 11, 2024, 8:43am UTC](https://discuss.huggingface.co/t/tokenization-compared-to-sentencepiece/106278/1 "2024-09-11T08:43:27Z")

</div>

Hi!  
Recently I tried to use microsoft/Phi-3-mini-4k-instruct tokenizer. I’ve found that results of the transformers tokenizer is different than the output of sentencepiece tokenizer. To give an example, here’s the sample code reproducing that issue:

```auto
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
formatted_text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
# '<|user|>\nWhat is capital city of France?<|end|>\n<|assistant|>\n'
tokenizer(formatted_text, return_tensors="pt")
# {'input_ids': tensor([[32010, 1724, 338, 7483, 4272, 310, 3444, 29973, 32007, 32001]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}

sp = spm.SentencePieceProcessor(model_file="...")
sp.encode(formatted_text)
# [32010, 13, 5618, 338, 7483, 4272, 310, 3444, 29973, 32007, 13, 32001, 13]

```

Do you know why transformers modify the original text? Is this correct behaviour? It removes newlines and add spaces (13 is a newline character, while 5618 → “What” and 1724 → “\_What”.