# SentencePiece to Tokenizers conversion

**URL:** <https://discuss.huggingface.co/t/sentencepiece-to-tokenizers-conversion/145756>\
**Category:** 🤗Tokenizers\
**Created:** [March 14, 2025, 10:18am UTC](https://discuss.huggingface.co/t/sentencepiece-to-tokenizers-conversion/145756 "2025-03-14T10:18:40Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![aaaksenova](https://avatars.discourse-cdn.com/v4/letter/a/a6a055/32.png) [@aaaksenova](https://discuss.huggingface.co/u/aaaksenova)\
**Post date:** [March 14, 2025, 10:18am UTC](https://discuss.huggingface.co/t/sentencepiece-to-tokenizers-conversion/145756/1 "2025-03-14T10:18:41Z")

</div>

Hey! I have trained a bpe model with sentencepiece library. I am converting it to tokenizers and have an issue with whitespace recognition

I am using this script to convert the tokenizer

```auto
import sentencepiece as spm
from transformers import convert_slow_tokenizer
from transformers import PreTrainedTokenizerFast

spm_tokenizer = spm.SentencePieceProcessor('tokeniser_training/tokenizer_fineweb_balanced_bpe_128000.model')
spm_tokenizer.vocab_file = 'tokeniser_training/tokenizer_fineweb_balanced_bpe_128000.model'
spm_converter = convert_slow_tokenizer.SpmConverter(spm_tokenizer)
converted = spm_converter.converted()
converted.save('converted.json')

tok = PreTrainedTokenizerFast.from_pretrained(pretrained_model_name_or_path="HuggingFaceTB/SmolLM-1.7B",
                                              tokenizer_file='converted.json', 
                                              clean_up_tokenization_spaces=False, pad_token='<|finetune_right_pad_id|>',
                                                unk_token='<unknown>', bos_token='<|start_of_sequence|>', 
                                                eos_token='<|end_of_sequence|>', 
                                                model_max_length=1024, 
                                                padding_side='right', truncation_side='right')
tok.save_pretrained('ConvertedTokenizer')

```

Generally tokenisation aligns well accept for whitespaces

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/b/e/be31d21f02a0592b0d6c917591b117b34dd23532.png)

Do es anyone have any idea what could be the issue? The desired behaviour of a new tokenizer is as seen in spm\_tokenizer