# Byte Level Tokenizer While Training

**URL:** <https://discuss.huggingface.co/t/byte-level-tokenizer-while-training/131009>\
**Category:** 🤗Tokenizers\
**Created:** [December 14, 2024, 1:27am UTC](https://discuss.huggingface.co/t/byte-level-tokenizer-while-training/131009 "2024-12-14T01:27:23Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![dinesh-bk](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dinesh-bk/32/37524_2.png) [@dinesh-bk](https://discuss.huggingface.co/u/dinesh-bk)\
**Post date:** [December 14, 2024, 1:27am UTC](https://discuss.huggingface.co/t/byte-level-tokenizer-while-training/131009/1 "2024-12-14T01:27:23Z")

</div>

Hi, I have trained the tokenizer using the model BPE and pre tokenizer as ByteLevel `tokenizer = Tokenizer(models.BPE(unk_token="[UNK]")) tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel() tokenizer.decoder = decoders.ByteLevel() `  
Now, my vocabulary is saved in bytes and tokenizer.tokenize give me output in bytes too which is obvious.  
`tokenizer.tokenize` output is

```auto
['Ġन',
 'à¥ĩ',
 'प',
 'ा',
 'ल',
 'à¥Ģ',
 'Ġà¤Ń',
 'ा',
 'ष',
 'ा',
 'म',
 'ा',
 'Ġय',
 'à¥ĭ',
 'Ġà¤ıà¤ķ',
 'Ġà¤īद',
 'ा',
 'हरण',
 'Ġह',
 'à¥ĭ।']

```

. Is there way to save my vocabulary in unicode character rather than bytes and show tokens in unicode characters too?