# Error with new tokenizers (URGENT!) **URL:** \ **Category:** 🤗Tokenizers\ **Created:** [December 16, 2020, 9:56am UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847 "2020-12-16T09:56:44Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![denocris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/denocris/32/2614_2.png) [@denocris](https://discuss.huggingface.co/u/denocris)\ **Post date:** [December 16, 2020, 9:56am UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/1 "2020-12-16T09:56:44Z")
Hi, recently all my pre-trained models undergo this error while loading their tokenizer: `Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece installed to convert a slow tokenizer to a fast one.` I tried to pip install sentencepiece but this does not solve the problem. Do you know any solution? (I am working on Google Colab) Note: In my humble opinion, changing so important things so fast can generate very dangerous problems. All my students (I teach DL stuff) and clients are stuck on my notebooks. I can understand that after a year a code can become outdated, but not just after two months. This requires a lot of maintenance work from my side! ---
**Author:** ![FL33TW00D](https://avatars.discourse-cdn.com/v4/letter/f/e0b2c6/32.png) [@FL33TW00D](https://discuss.huggingface.co/u/FL33TW00D)\ **Post date:** [December 16, 2020, 10:17am UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/2 "2020-12-16T10:17:26Z")
There were some breaking changes in the V4 release, please find the details here: > **[Release Transformers v4.0.0: Fast tokenizers, model outputs, file...](https://github.com/huggingface/transformers/releases/tag/v4.0.0)** > > Transformers v4.0.0-rc-1: Fast tokenizers, model outputs, file reorganization > Breaking changes since v3.x > Version v4.0.0 introduces several breaking changes that were necessary. > 1. AutoTokenizers a... ---
**Author:** ![denocris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/denocris/32/2614_2.png) [@denocris](https://discuss.huggingface.co/u/denocris)\ **Post date:** [December 16, 2020, 10:26am UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/3 "2020-12-16T10:26:08Z")
Thank you so much @FL33TW00D. Two steps were required. 1. latest pip version 20.3.3 (On Colab I had installed by default 19 and something); 2. use\_fast = False. Working a lot on Italian or personal custom models, all of them were disabled all of a sudden! ---
**Author:** ![Ogayo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ogayo/32/16787_2.png) [@Ogayo](https://discuss.huggingface.co/u/Ogayo)\ **Post date:** [December 16, 2020, 12:43pm UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/4 "2020-12-16T12:43:54Z")
Hi @denocris I am facing the same problem. I am a student and I depend on this for my presentation tomorrow. I am also using Google Colab. Can you please explain in steps how to fix this sentencpiece problem? Thanks in advance ---
**Author:** ![Ogayo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ogayo/32/16787_2.png) [@Ogayo](https://discuss.huggingface.co/u/Ogayo)\ **Post date:** [December 16, 2020, 1:03pm UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/5 "2020-12-16T13:03:50Z")
> [@denocris](#): > > - latest pip version 20.3.3 (On Colab I had installed by default 19 and something); I resolved it. 1. Uninstalled transformers 2. Installed transformers sentencepiece like this : **!pip install --no-cache-dir transformers sentencepiece** 3. Use\_fast= False like this: **tokenizer = AutoTokenizer.from\_pretrained(“XXXXX”, use\_fast=False)** ---
**Author:** ![denocris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/denocris/32/2614_2.png) [@denocris](https://discuss.huggingface.co/u/denocris)\ **Post date:** [December 16, 2020, 4:41pm UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/6 "2020-12-16T16:41:26Z")
Sorry @Ogayo, I have just read. I am happy you solved it. It is quite annoying to have these kind of issues from one day to another. I had this error during a live presentation. A couple of days before the notebook was working well! ---
**Author:** ![kroshan](https://avatars.discourse-cdn.com/v4/letter/k/e79b87/32.png) [@kroshan](https://discuss.huggingface.co/u/kroshan)\ **Post date:** [April 5, 2022, 7:23pm UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/7 "2022-04-05T19:23:00Z")
Thanku @Ogayo ---
**Author:** ![evegarcianz](https://avatars.discourse-cdn.com/v4/letter/e/c89c15/32.png) [@evegarcianz](https://discuss.huggingface.co/u/evegarcianz)\ **Post date:** [February 15, 2023, 8:25pm UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/8 "2023-02-15T20:25:13Z")
Thank you @Ogayo !! ---
**Author:** ![zeelthumar-04](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/zeelthumar-04/32/16772_2.png) [@zeelthumar-04](https://discuss.huggingface.co/u/zeelthumar-04)\ **Post date:** [June 12, 2023, 10:27am UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/9 "2023-06-12T10:27:42Z")
THANKS A LOT @Ogayo ---
**Author:** ![ashu3984](https://avatars.discourse-cdn.com/v4/letter/a/e47774/32.png) [@ashu3984](https://discuss.huggingface.co/u/ashu3984)\ **Post date:** [October 2, 2023, 6:14pm UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/10 "2023-10-02T18:14:52Z")
Thank you very much @Ogayo. ---
**Author:** ![VishNikhil](https://avatars.discourse-cdn.com/v4/letter/v/dc4da7/32.png) [@VishNikhil](https://discuss.huggingface.co/u/VishNikhil)\ **Post date:** [November 22, 2023, 1:15pm UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/11 "2023-11-22T13:15:37Z")
Hi @Ogayo ,im getting the same error after resolving the following steps also what should i do ? ---
**Author:** ![borat123](https://avatars.discourse-cdn.com/v4/letter/b/c77e96/32.png) [@borat123](https://discuss.huggingface.co/u/borat123)\ **Post date:** [March 6, 2024, 12:00pm UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/12 "2024-03-06T12:00:29Z")
> [@Ogayo](#): > > pip install --no-cache-dir transformers Same problem here. Solved it using: `pip install sentencepiece` ---
**Author:** ![Hope2000](https://avatars.discourse-cdn.com/v4/letter/h/ac8455/32.png) [@Hope2000](https://discuss.huggingface.co/u/Hope2000)\ **Post date:** [April 4, 2024, 7:42am UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/13 "2024-04-04T07:42:41Z")
I tried using `use_fast=False` but got the following error: ```auto SyntaxError: keyword argument repeated: use_fast ``` I think it’s deprecated now! but the installation part works, so thank you. ---
**Author:** ![MahadA](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@MahadA](https://discuss.huggingface.co/u/MahadA)\ **Post date:** [May 15, 2024, 9:40am UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/14 "2024-05-15T09:40:59Z")
Hello Denocris Hope you are doing well i am getting this error * * * KeyError Traceback (most recent call last) in \() 1 from transformers import AutoTokenizer, AutoModel 2 ----\> 3 tokenizer = AutoTokenizer.from\_pretrained(“musadac/vilanocr-multi-medical”, use\_fast=False) 1 frames /usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto\_factory.py in **getitem** (self, key) 740 model\_name = self.\_model\_mapping[mtype] 741 return self.\_load\_attr\_from\_module(mtype, model\_name) → 742 raise KeyError(key) 743 744 def \_load\_attr\_from\_module(self, model\_type, attr): KeyError: \ ---
**Author:** ![joyz29](https://avatars.discourse-cdn.com/v4/letter/j/c77e96/32.png) [@joyz29](https://discuss.huggingface.co/u/joyz29)\ **Post date:** [July 19, 2024, 9:48am UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/15 "2024-07-19T09:48:23Z")
> [@Ogayo](#): > > !pip install --no-cache-dir transformers sentencepiece it works! thankx sooo much ---
**Author:** ![Kaxder23](https://avatars.discourse-cdn.com/v4/letter/k/c5a1d2/32.png) [@Kaxder23](https://discuss.huggingface.co/u/Kaxder23)\ **Post date:** [July 20, 2024, 9:59am UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/16 "2024-07-20T09:59:58Z")
Thank you very much for this… ---
**Author:** ![Kaxder23](https://avatars.discourse-cdn.com/v4/letter/k/c5a1d2/32.png) [@Kaxder23](https://discuss.huggingface.co/u/Kaxder23)\ **Post date:** [July 22, 2024, 7:58am UTC](https://discuss.huggingface.co/t/error-with-new-tokenizers-urgent/2847/17 "2024-07-22T07:58:12Z")
Hi, I’ve faced the same issue while integrating it into a [calculator-based website](https://mauricettecalculette.fr/). Installing transformers with sentencepiece and restarting the runtime worked for me. Try this: “pip install transformers sentencepiece”