# Tokenizer unigram tutorial encode\_word function question

**URL:** <https://discuss.huggingface.co/t/tokenizer-unigram-tutorial-encode-word-function-question/86224>\
**Category:** Beginners\
**Created:** [May 11, 2024, 9:30am UTC](https://discuss.huggingface.co/t/tokenizer-unigram-tutorial-encode-word-function-question/86224 "2024-05-11T09:30:59Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nevermetyou](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nevermetyou/32/21618_2.png) [@Nevermetyou](https://discuss.huggingface.co/u/Nevermetyou)\
**Post date:** [May 11, 2024, 9:30am UTC](https://discuss.huggingface.co/t/tokenizer-unigram-tutorial-encode-word-function-question/86224/1 "2024-05-11T09:30:59Z")

</div>

Hi

I just finish reading this tutorial on Unigram.

> **[Unigram tokenization - Hugging Face NLP Course](https://huggingface.co/learn/nlp-course/chapter6/7?fw=pt#implementing-unigram)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

And I have a question about this line in function `encode_word()`

```auto
best_segmentations = [{"start": 0, "score": 1}] + [
        {"start": None, "score": None} for _ in range(len(word))
    ]

```

As per my understanding, the score inside dictionary of the first list should be log(1) not 1 ???

Because in this line  
`score = model[token] + best_score_at_start`  
we are summing the log of probability.

So I suspect that `[{"start": 0, "score": 1}]` should be `[{"start": 0, "score": 0}]`

Can someone clarify me this matter?

Thanks