# Adding new tokens to a BERT tokenizer - Getting ValueError

**URL:** <https://discuss.huggingface.co/t/adding-new-tokens-to-a-bert-tokenizer-getting-valueerror/9253>\
**Category:** 🤗Tokenizers\
**Created:** [August 15, 2021, 10:49am UTC](https://discuss.huggingface.co/t/adding-new-tokens-to-a-bert-tokenizer-getting-valueerror/9253 "2021-08-15T10:49:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![kaankork](https://avatars.discourse-cdn.com/v4/letter/k/e9bcb4/32.png) [@kaankork](https://discuss.huggingface.co/u/kaankork)\
**Post date:** [August 15, 2021, 10:49am UTC](https://discuss.huggingface.co/t/adding-new-tokens-to-a-bert-tokenizer-getting-valueerror/9253/1 "2021-08-15T10:49:52Z")

</div>

I have a Python list named `unique_list` that contains new words that will be added to my tokenizer using` tokenizer.add_tokens`. However, when I run my code I’m getting the following error:

```auto
File "/home/kaan/anaconda3/envs/env_backup/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 937, in add_tokens
    if not new_tokens:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

```

When I tested with a test array that contains 10 random words, it worked fine but the larger unique\_list is causing a problem.

What am I doing wrong here?

---

<div class="post-metadata">

**Author:** ![impiyush](https://avatars.discourse-cdn.com/v4/letter/i/c6cbf5/32.png) [@impiyush](https://discuss.huggingface.co/u/impiyush)\
**Post date:** [December 3, 2021, 10:16pm UTC](https://discuss.huggingface.co/t/adding-new-tokens-to-a-bert-tokenizer-getting-valueerror/9253/2 "2021-12-03T22:16:00Z")

</div>

Not sure if you already solved this issue, but I stumbled upon it today.

Looking closely at the error message, looks like the `add_tokens()` method expects the `new_tokens` passed to it as a python list rather than an numpy array. Converting `new_tokens` from numpy array to a list and then passing it resolved the issue.

`added_tokens = tokenizer.add_tokens(new_tokens.tolist())`

---

<div class="post-metadata">

**Author:** ![toyl](https://avatars.discourse-cdn.com/v4/letter/t/73ab20/32.png) [@toyl](https://discuss.huggingface.co/u/toyl)\
**Post date:** [January 16, 2022, 7:06pm UTC](https://discuss.huggingface.co/t/adding-new-tokens-to-a-bert-tokenizer-getting-valueerror/9253/4 "2022-01-16T19:06:31Z")

</div>

excuse me if I need to solve the problem of not finding the word after tokenizing using BERT … can I use your solution