# Issue with sentencepiece tokenizer

**URL:** <https://discuss.huggingface.co/t/issue-with-sentencepiece-tokenizer/20840>\
**Category:** 🤗Transformers\
**Created:** [July 25, 2022, 2:03pm UTC](https://discuss.huggingface.co/t/issue-with-sentencepiece-tokenizer/20840 "2022-07-25T14:03:05Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ghpkishore](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@ghpkishore](https://discuss.huggingface.co/u/ghpkishore)\
**Post date:** [July 25, 2022, 2:03pm UTC](https://discuss.huggingface.co/t/issue-with-sentencepiece-tokenizer/20840/1 "2022-07-25T14:03:05Z")

</div>

Hi, I am using deepset haystack for question generation and get the following error.

> Blockquote  
> File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/haystack/nodes/question\_generator/question\_generator.py”, line 59, in **init**  
> self.tokenizer = AutoTokenizer.from\_pretrained(model\_name\_or\_path)  
> File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/models/auto/tokenization\_auto.py”, line 597, in from\_pretrained  
> return tokenizer\_class\_fast.from\_pretrained(pretrained\_model\_name\_or\_path, \*inputs, \*\*kwargs)  
> File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/tokenization\_utils\_base.py”, line 1783, in from\_pretrained  
> return cls.\_from\_pretrained(  
> File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/tokenization\_utils\_base.py”, line 1928, in \_from\_pretrained  
> tokenizer = cls(\*init\_inputs, \*\*init\_kwargs)  
> File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/models/t5/tokenization\_t5\_fast.py”, line 134, in **init**  
> super(). **init** (  
> File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/tokenization\_utils\_fast.py”, line 113, in **init**  
> fast\_tokenizer = convert\_slow\_tokenizer(slow\_tokenizer)  
> File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/convert\_slow\_tokenizer.py”, line 1077, in convert\_slow\_tokenizer  
> return converter\_class(transformer\_tokenizer).converted()  
> File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/convert\_slow\_tokenizer.py”, line 426, in **init**  
> from .utils import sentencepiece\_model\_pb2 as model\_pb2  
> File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/utils/sentencepiece\_model\_pb2.py”, line 92, in   
> \_descriptor.EnumValueDescriptor(  
> File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/google/protobuf/descriptor.py”, line 755, in **new**  
> \_message.Message.\_CheckCalledFromGeneratedFile()  
> TypeError: Descriptors cannot not be created directly.  
> If this call came from a \_pb2.py file, your generated code is out of date and must be regenerated with protoc \>= 3.19.0.  
> If you cannot immediately regenerate your protos, some other possible workarounds are:

1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL\_BUFFERS\_PYTHON\_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

Please let me know how to resolve this.

---

<div class="post-metadata">

**Author:** ![sjrhuschlee](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sjrhuschlee/32/24023_2.png) [@sjrhuschlee](https://discuss.huggingface.co/u/sjrhuschlee)\
**Post date:** [July 25, 2022, 2:21pm UTC](https://discuss.huggingface.co/t/issue-with-sentencepiece-tokenizer/20840/2 "2022-07-25T14:21:22Z")

</div>

It looks like you need to install a different version of `protobuf`. Try running `pip install protobuf==3.20.*` in your python environment and then rerun your code.

---

<div class="post-metadata">

**Author:** ![ghpkishore](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@ghpkishore](https://discuss.huggingface.co/u/ghpkishore)\
**Post date:** [July 25, 2022, 2:28pm UTC](https://discuss.huggingface.co/t/issue-with-sentencepiece-tokenizer/20840/3 "2022-07-25T14:28:51Z")

</div>

> [@sjrhuschlee](#):
>
> pip install protobuf==3.20.\*

Generating the questions now, but it throws the following error:

```auto
[nltk_data] Error loading punkt: <urlopen error [SSL:
[nltk_data] CERTIFICATE_VERIFY_FAILED] certificate verify failed:
[nltk_data] unable to get local issuer certificate (_ssl.c:997)>

```