# Language model for wav2vec2.0 decoding **URL:** \ **Category:** Models\ **Created:** [March 16, 2021, 6:02am UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434 "2021-03-16T06:02:17Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![EmreOzkose](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/emreozkose/32/18399_2.png) [@EmreOzkose](https://discuss.huggingface.co/u/EmreOzkose)\ **Post date:** [March 16, 2021, 6:02am UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/1 "2021-03-16T06:02:17Z")
Hello, I implemented [wav2vec2.0 code](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2) and a language model is not used for decoding. How can I add a language model (let’s say a language model which is trained with KenLM) for decoding @patrickvonplaten ? thanks in advance. Note: I also opened an [issue](https://github.com/huggingface/transformers/issues/10716), but redirected here. ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [March 16, 2021, 2:31pm UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/2 "2021-03-16T14:31:00Z")
Hey Emre! Yeah good question - we currently don’t support evaluating with a language model, but we plan on adding this functionality soon! It’s sadly not that trivial to decode a CTC model with a language model. I’ll try to keep you posted for updates here! ---
**Author:** ![andersgb1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/andersgb1/32/1435_2.png) [@andersgb1](https://discuss.huggingface.co/u/andersgb1)\ **Post date:** [April 17, 2021, 1:03pm UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/3 "2021-04-17T13:03:32Z")
Assuming that one has a kenlm model already, am I wrong to assume that’s it’s just a matter of giving the wav2vec2 output logits as argument to the ctcdecode main function, exemplified here: [GitHub - parlance/ctcdecode: PyTorch CTC Decoder bindings](https://github.com/parlance/ctcdecode#how-to-use)? Or is there more to it than that? ---
**Author:** ![sandersbud4](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sandersbud4/32/1487_2.png) [@sandersbud4](https://discuss.huggingface.co/u/sandersbud4)\ **Post date:** [April 19, 2021, 7:56am UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/4 "2021-04-19T07:56:27Z")
@EmreOzkose Good question i think. But i don’t start this as professional level. I’m currently searching on this. 🙂 ---
**Author:** ![Wikidepia](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wikidepia/32/1099_2.png) [@Wikidepia](https://discuss.huggingface.co/u/Wikidepia)\ **Post date:** [April 22, 2021, 10:41pm UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/5 "2021-04-22T22:41:03Z")
Hi all, I’ve been experimenting kenlm with wav2vec2 here is the [notebok](https://github.com/Wikidepia/wav2vec2-indonesian/blob/master/notebooks/kenlm-wav2vec2.ipynb) I dont know if this is a proper implementation, but it works! I also still need to cleanup some stuff like vocab & other thing. ---
**Author:** ![SamuelAzran](https://avatars.discourse-cdn.com/v4/letter/s/8797f3/32.png) [@SamuelAzran](https://discuss.huggingface.co/u/SamuelAzran)\ **Post date:** [April 23, 2021, 7:39am UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/6 "2021-04-23T07:39:49Z")
@Wikidepia Can you share how much it improved your WER score? Also, did you tried character level LM as well? ---
**Author:** ![Wikidepia](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wikidepia/32/1099_2.png) [@Wikidepia](https://discuss.huggingface.co/u/Wikidepia)\ **Post date:** [April 24, 2021, 8:43am UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/7 "2021-04-24T08:43:32Z")
It improved from 14.2 to 9.2. I haven’t tried character level LM 😅 ---
**Author:** ![Edresson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/edresson/32/1550_2.png) [@Edresson](https://discuss.huggingface.co/u/Edresson)\ **Post date:** [April 24, 2021, 4:54pm UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/8 "2021-04-24T16:54:28Z")
I added support for KenLM using the flashlight library here: [Wav2Vec-Wrapper/test.py at main · Edresson/Wav2Vec-Wrapper · GitHub](https://github.com/Edresson/Wav2Vec-Wrapper/blob/main/test.py) It supports the use of the binary file instead of the arpa and it is also possible to restrict the model’s vocabulary. ---
**Author:** ![EmreOzkose](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/emreozkose/32/18399_2.png) [@EmreOzkose](https://discuss.huggingface.co/u/EmreOzkose)\ **Post date:** [April 26, 2021, 5:40am UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/9 "2021-04-26T05:40:00Z")
Thank you @Wikidepia and @Edresson. I will check out. ---
**Author:** ![Beau](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/beau/32/1634_2.png) [@Beau](https://discuss.huggingface.co/u/Beau)\ **Post date:** [May 10, 2021, 9:44pm UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/10 "2021-05-10T21:44:06Z")
Hi Patrick! Any news on language model evaluation support? ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [May 16, 2021, 6:39pm UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/11 "2021-05-16T18:39:32Z")
See [Added Feature: Prefix decoding for wav2vec2 models by deepang17 · Pull Request #11606 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/pull/11606) ---
**Author:** ![jolurf](https://avatars.discourse-cdn.com/v4/letter/j/34f0e0/32.png) [@jolurf](https://discuss.huggingface.co/u/jolurf)\ **Post date:** [May 27, 2021, 8:19pm UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/12 "2021-05-27T20:19:40Z")
Wiki, when I apply your code, it predicts only spaces and -. Is there any reason for it? ---
**Author:** ![ChristophBensch](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/christophbensch/32/3003_2.png) [@ChristophBensch](https://discuss.huggingface.co/u/ChristophBensch)\ **Post date:** [June 15, 2021, 10:10pm UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/13 "2021-06-15T22:10:16Z")
@jolurf you can also use this decoder ([GitHub - parlance/ctcdecode: PyTorch CTC Decoder bindings](https://github.com/parlance/ctcdecode)). Take the labels from your tokenizer and create a n-gram language model with KenLM. After that you can feed the logits from your Wav2Vec2 model into the decoder. @patrickvonplaten Are there any updates on the transformer language model? Voidful suggested to combine the wav2vec2 probabilities with those of the gpt2 model: ( [huggingface\_notebook/xlsr\_gpt.ipynb at main · voidful/huggingface\_notebook · GitHub](https://github.com/voidful/huggingface_notebook/blob/main/xlsr_gpt.ipynb) ) However, the CTC Vocab seems to match the GPT Vocab. Unfortunately, this is not the case in English. Is there already a solution? ---
**Author:** ![DewiBrynJones](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dewibrynjones/32/3696_2.png) [@DewiBrynJones](https://discuss.huggingface.co/u/DewiBrynJones)\ **Post date:** [June 17, 2021, 2:09pm UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/14 "2021-06-17T14:09:46Z")
If this discussion is still ongoing, then there is a pull request [Added Feature: Prefix decoding for wav2vec2 models by deepang17 · Pull Request #11606 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/pull/11606) currently open, and as @ChristophBensch mentions a means of using KenLM from [GitHub - parlance/ctcdecode: PyTorch CTC Decoder bindings](https://github.com/parlance/ctcdecode). We have an example of this at [GitHub - techiaith/docker-wav2vec2-xlsr-ft-cy: Hyfforddi modelau adnabod lleferydd Cymraeg wav2vec2 a KenLM a'u darparu drwy weinydd gwasanaeth API // Train wav2vec2 and KenLM models for Welsh language speech recognition and/or provide via a simple API server.](https://github.com/techiaith/docker-wav2vec2-xlsr-ft-cy) that’s reduced our WER score for Welsh from 25% to 15%. Since our scripts use HuggingFace’s OSCAR dataset, they should be easily adaptable to train and optimize LMs for other lesser resourced languages as well. ---
**Author:** ![othiele](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/othiele/32/19711_2.png) [@othiele](https://discuss.huggingface.co/u/othiele)\ **Post date:** [June 24, 2021, 8:11am UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/15 "2021-06-24T08:11:07Z")
Thanks @DewiBrynJones for the implementation, love the idea to have readmes in your local language and reference an English version 🙂 ---
**Author:** ![OthmaneJ](https://avatars.discourse-cdn.com/v4/letter/o/eb9ed0/32.png) [@OthmaneJ](https://discuss.huggingface.co/u/OthmaneJ)\ **Post date:** [July 1, 2021, 8:09am UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/16 "2021-07-01T08:09:13Z")
Hi all! As advised by @andersgb1 I used a kenlm n-gram language model on top of a distilled wav2vec2 that I trained and it improved my WER (26 → 12.6). If you guys are interested here’s the notebook (executes seamlessly on colab) [OthmaneJ/distil-wav2vec2 · Hugging Face](https://huggingface.co/OthmaneJ/distil-wav2vec2) ---
**Author:** ![agemagician](https://avatars.discourse-cdn.com/v4/letter/a/d07c76/32.png) [@agemagician](https://discuss.huggingface.co/u/agemagician)\ **Post date:** [July 5, 2021, 2:58pm UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/17 "2021-07-05T14:58:30Z")
@OthmaneJ Could you please share the code you used for distilling wav2vec2? ---
**Author:** ![junxtjx](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/junxtjx/32/2464_2.png) [@junxtjx](https://discuss.huggingface.co/u/junxtjx)\ **Post date:** [July 5, 2021, 3:23pm UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/18 "2021-07-05T15:23:36Z")
So to use the wav2vec2 with gpt2 for English, would we have to just match the vocab used in the wav2vec2 with the vocab used in the gpt2? ---
**Author:** ![youssefav](https://avatars.discourse-cdn.com/v4/letter/y/5daacb/32.png) [@youssefav](https://discuss.huggingface.co/u/youssefav)\ **Post date:** [July 7, 2021, 10:23am UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/19 "2021-07-07T10:23:49Z")
Is integrating an LM for wav2vec2 basically pointless now with the release of HuBERT? Which if I understand correctly is both an audio and language model at the same time? [facebook/hubert-xlarge-ll60k · Hugging Face](https://huggingface.co/facebook/hubert-xlarge-ll60k) I’m trying to achieve sub-5% (surpassing human performance) WER, but I don’t know if after I fine-tune this Hubert on my own data it will achieve that or not, because I’m not sure about the language model thing. Does it also need an integration with a language model to actually make it perform well? ---
**Author:** ![ThomasG](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomasg/32/2445_2.png) [@ThomasG](https://discuss.huggingface.co/u/ThomasG)\ **Post date:** [September 13, 2021, 7:21pm UTC](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434/20 "2021-09-13T19:21:21Z")
Can I create a character-level LM with KenLM? [Next page](https://discuss.huggingface.co/t/language-model-for-wav2vec2-0-decoding/4434.md?page=2)