# Ġ token inserted by ByteLevelBPETokenizer

**URL:** <https://discuss.huggingface.co/t/g-token-inserted-by-bytelevelbpetokenizer/60678>\
**Category:** 🤗Transformers\
**Created:** [November 1, 2023, 8:41am UTC](https://discuss.huggingface.co/t/g-token-inserted-by-bytelevelbpetokenizer/60678 "2023-11-01T08:41:03Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![innomadic](https://avatars.discourse-cdn.com/v4/letter/i/b4bc9f/32.png) [@innomadic](https://discuss.huggingface.co/u/innomadic)\
**Post date:** [November 1, 2023, 8:41am UTC](https://discuss.huggingface.co/t/g-token-inserted-by-bytelevelbpetokenizer/60678/1 "2023-11-01T08:41:03Z")

</div>

In this example notebook [How to train a new language model from scratch using Transformers and Tokenizers](https://colab.research.google.com/github/huggingface/blog/blob/main/notebooks/01_how_to_train.ipynb), i notice that after the encoding step, some new characters are introduced to some of the tokens:

```auto
tokenizer.encode("Mi estas Julien.").tokens

```

Results in

```auto
['<s>', 'Mi', 'Ġestas', 'ĠJuli', 'en', '.', '</s>']

```

What is the significance of these G’s with marks over them? I thought they could represent something like a word that is continuing, or that is broken apart by the tokenizer, but that doesn’t appear to be true in this example.

I ask in part because when I try to train this on another language, I end up with these same G’s but they are standalone tokens

```auto
tokenizer.encode("Waan ku salaamayaa.").tokens
['<s>', 'W', 'aan', 'Ġ', 'ku', 'Ġ', 'sal', 'aam', 'ay', 'aa', '.', '</s>']

```

Because I am not sure of there meaning, I am not sure if this is a problem, and I am not sure if it is a problem that they are separate tokens in my example, but they are only at the beginning of the tokens in the sample code.

Edit: I can see from researching some that this represents a space. But why does it seem to be attached to other tokens in the example, but it stands alone in my project?