# Russian ASR: Fine-tuning Wav2Vec2 **URL:** \ **Category:** Languages at Hugging Face\ **Created:** [March 20, 2021, 7:58am UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770 "2021-03-20T07:58:24Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![arkadyark](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arkadyark/32/3421_2.png) [@arkadyark](https://discuss.huggingface.co/u/arkadyark)\ **Post date:** [March 20, 2021, 7:58am UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/1 "2021-03-20T07:58:24Z")
Hello everybody! Creating a thread to organize the work on Russian ASR Data: - Common voice (111 hours validated) - CSS10 Russian: Single Speaker Speech Dataset (small, 440 utterances, available at [Russian Single Speaker Speech Dataset | Kaggle](https://www.kaggle.com/bryanpark/russian-single-speaker-speech-dataset)) - Open STT ([GitHub - snakers4/open\_stt: Open STT](https://github.com/snakers4/open_stt/) very large, ~20k hours, multi-domain, probably too use in full) Progress: So far, @gorodecki has begun training on a subset of the common voice dataset. I tried briefly to run on the whole dataset on colab, but quickly ran out of memory so I’ll have to revisit that. Another thought I had that could be a stretch goal was to use this Russian model as a base from which to further fine-tune on similar slavic languages/dialects that may be lower resource. I was thinking of Belarusian, Ukrainian, Kazakh Russian, any others? Tagging those involved so far (tag anybody else who would like to join us as well!): @gorodecki @vladdy ---
**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](https://discuss.huggingface.co/u/gorodecki)\ **Post date:** [March 20, 2021, 8:38am UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/2 "2021-03-20T08:38:52Z")
Hello! My result is 0,38 WER on part of common voice. ---
**Author:** ![anton-l](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anton-l/32/2771_2.png) [@anton-l](https://discuss.huggingface.co/u/anton-l)\ **Post date:** [March 20, 2021, 7:41pm UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/3 "2021-03-20T19:41:37Z")
Hi everyone! Let’s see what we can do in a week 🚀 ---
**Author:** ![arkadyark](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arkadyark/32/3421_2.png) [@arkadyark](https://discuss.huggingface.co/u/arkadyark)\ **Post date:** [March 21, 2021, 7:56pm UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/4 "2021-03-21T19:56:57Z")
Nice, that’s a great starting point! I think I have access to a GPU machine with more disk space, so I’ll try overnight tonight to see if I can download the whole dataset onto there and see what results I get with the full training set. How long did training on 10% take for you with Colab? ---
**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](https://discuss.huggingface.co/u/gorodecki)\ **Post date:** [March 21, 2021, 10:12pm UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/5 "2021-03-21T22:12:02Z")
I’m use local machine with T4. It takes approximately 4 hours on one GPU. Unfortunately, training cannot be started on 4 GPUs at parallel mode. ---
**Author:** ![arkadyark](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arkadyark/32/3421_2.png) [@arkadyark](https://discuss.huggingface.co/u/arkadyark)\ **Post date:** [March 21, 2021, 10:41pm UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/6 "2021-03-21T22:41:10Z")
I see - I have access to a machine with 2 GPUs (P100) but also ran into an error trying to run it with DistributedDataParallel. I’m trying to run it now with just 1GPU with the full common voice dataset. Did you also get this same error? “RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn’t able to locate the output tensors in the return value of your module’s `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).” ---
**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](https://discuss.huggingface.co/u/gorodecki)\ **Post date:** [March 22, 2021, 12:24pm UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/7 "2021-03-22T12:24:37Z")
No, my error is “CUDA not enough memory” I run train jupyter notebook. ---
**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](https://discuss.huggingface.co/u/gorodecki)\ **Post date:** [March 22, 2021, 1:18pm UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/8 "2021-03-22T13:18:31Z")
![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/8/8a7c7c11537eaa5795e8948d95e7fd33579db6ee.png) Can you view this parametr in your `training_args` ? ---
**Author:** ![arkadyark](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arkadyark/32/3421_2.png) [@arkadyark](https://discuss.huggingface.co/u/arkadyark)\ **Post date:** [March 22, 2021, 10:57pm UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/9 "2021-03-22T22:57:06Z")
I’ll give that a try! In the meantime - my training run on the whole dataset on a single GPU has just about finished running for 5 epochs, finishing with a WER on the validation set of about 39%. I’d hoped it would do better than your smaller split, but it seems like it didn’t. The WER does still seem to be decreasing, although slowly, so maybe more training time might help but I’m not sure. This first model is a good starting point to iterate on. ---
**Author:** ![arkadyark](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arkadyark/32/3421_2.png) [@arkadyark](https://discuss.huggingface.co/u/arkadyark)\ **Post date:** [March 23, 2021, 7:03am UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/10 "2021-03-23T07:03:06Z")
Ok, a couple more updates: - When I ran with 2GPUs and had an error, training\_args.parallel\_mode was equal to ParallelMode.DISTRIBUTED, so I’m not sure what’s going wrong there - I finished running training on common\_voice for 5 epochs on a single GPU, my final WER on the evaluation set remained around 0.39. In the end, training took about 19 hours. As a next step, I may try to load in a portion of the OpenSTT dataset and add this to the training set to see if it improves performance. Any other ideas? ---
**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](https://discuss.huggingface.co/u/gorodecki)\ **Post date:** [March 23, 2021, 3:43pm UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/11 "2021-03-23T15:43:39Z")
Good work! @anton-l create this script [anton-l/wav2vec2-large-xlsr-53-russian · Hugging Face](https://huggingface.co/anton-l/wav2vec2-large-xlsr-53-russian). You test `sentence_clean` for evaluate! ёжик == ежик==:hedgehog: in russian language 😊 ---
**Author:** ![arkadyark](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arkadyark/32/3421_2.png) [@arkadyark](https://discuss.huggingface.co/u/arkadyark)\ **Post date:** [March 24, 2021, 10:14am UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/12 "2021-03-24T10:14:17Z")
Thanks for sharing, I’ll evaluate on the test set and share my results. @anton-l I see you got a WER of 22.4% - can you share more about the model you trained? Was it trained with the whole common-voice dataset? ---
**Author:** ![arkadyark](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arkadyark/32/3421_2.png) [@arkadyark](https://discuss.huggingface.co/u/arkadyark)\ **Post date:** [March 24, 2021, 10:38am UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/13 "2021-03-24T10:38:20Z")
Running the evaluation script on my model trained on all of common voice, I get a WER on the test set of 30.6 ---
**Author:** ![anton-l](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anton-l/32/2771_2.png) [@anton-l](https://discuss.huggingface.co/u/anton-l)\ **Post date:** [March 26, 2021, 10:25am UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/14 "2021-03-26T10:25:18Z")
I tried augmenting the Ukrainian dataset by phonetically translating the Russian one (2+ times larger) using some crude transliteration rules for passport names. Didn’t help the WER even one bit, but it was fun to try nevertheless 😂 ![translation](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/d2b31317e2b1ce74cdab5d1b4518fc8336ba1f49.png) ---
**Author:** ![anton-l](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anton-l/32/2771_2.png) [@anton-l](https://discuss.huggingface.co/u/anton-l)\ **Post date:** [March 26, 2021, 10:32am UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/15 "2021-03-26T10:32:57Z")
Also, Open STT turned out to be too noisy (both in term of background sounds and annotation) to have any positive effect on Common Voice test metrics, so probably not worth it just yet. ---
**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](https://discuss.huggingface.co/u/gorodecki)\ **Post date:** [March 26, 2021, 11:29am UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/16 "2021-03-26T11:29:03Z")
I think before training on noisy data, you need to get a WER of about 10% in order for the probabilities to be high. ---
**Author:** ![gorodecki](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@gorodecki](https://discuss.huggingface.co/u/gorodecki)\ **Post date:** [March 27, 2021, 8:14pm UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/17 "2021-03-27T20:14:57Z")
🙂 What metric updates are there? What new experiments have been successful? My result is 29% WER on CommonVoice. ---
**Author:** ![arkadyark](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arkadyark/32/3421_2.png) [@arkadyark](https://discuss.huggingface.co/u/arkadyark)\ **Post date:** [March 28, 2021, 9:37pm UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/18 "2021-03-28T21:37:50Z")
Hi! I got similar results, and didn’t have a chance to train more models after that - it seems like @anton-l 's model that got 22% was the best one! ---
**Author:** ![anton-l](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anton-l/32/2771_2.png) [@anton-l](https://discuss.huggingface.co/u/anton-l)\ **Post date:** [March 29, 2021, 7:11am UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/19 "2021-03-29T07:11:59Z")
After longer training (60 epochs) and augmentations I was able to squeeze out 17.39 WER! 🙀 Check out the details on Slack ---
**Author:** ![arkadyark](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arkadyark/32/3421_2.png) [@arkadyark](https://discuss.huggingface.co/u/arkadyark)\ **Post date:** [May 21, 2021, 3:43pm UTC](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770/20 "2021-05-21T15:43:16Z")
Just found another new corpus for Russian ASR here, that was able to get as low as 8% WER on Common Voice test set. Would be cool to try to add this together with the dataset we trained on, and see if we can beat that! [https://github.com/sberdevices/golos](https://github.com/sberdevices/golos) [Next page](https://discuss.huggingface.co/t/russian-asr-fine-tuning-wav2vec2/4770.md?page=2)