# Build a language detector **URL:** \ **Category:** 🤗 Course Projects\ **Created:** [November 9, 2021, 7:11pm UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520 "2021-11-09T19:11:58Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [November 9, 2021, 7:11pm UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520/1 "2021-11-09T19:11:58Z")
> 👋 Please read the topic category [description](https://discuss.huggingface.co/t/about-the-course-projects-category/11510) to understand what this is all about ### Description For many online applications it is not known in advance what language an end-user will communicate in. The goal of this project is to build a system that can automatically predict the language a text is written in. ### Model(s) There are a few popular multilingual models that you can start with: - [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) - [bert-base-multilingual-uncased](https://huggingface.co/bert-base-multilingual-uncased) ### Datasets There are quite a few [multilingual datasets](https://huggingface.co/datasets?multilinguality=multilinguality:multilingual&sort=downloads) available on the Hub. Many of these have a “language” field that could be used as a target for the model to predict. ### Challenges This project will likely require you to combine several datasets together to gain enough coverage of many languages. ### Desired project outcomes - Create a Streamlit or Gradio app on [🤗 Spaces](https://huggingface.co/spaces) that can predict the language of a piece of text provided by an end-user - Don’t forget to push all your models and datasets to the Hub so others can build on them! ### Additional resources A good baseline to compare your model against is the Python `langid` [library](https://github.com/saffsd/langid.py) ### Discord channel To chat and organise with other people interested in this project, head over to our [Discord](http://xwo7x.mjt.lu/lnk/CAAAAkdkZwAAAAAAAAAAALG-MTEAAYCr4H8AAAAAAAyFyQBhjpnimM4rcE5DT5ibCUtRT96tWwAMh8Y/15/U6BYNMNDksyyGG4jgtzPKQ/aHR0cHM6Ly9odWdnaW5nZmFjZS5jby9qb2luL2Rpc2NvcmQ) and: - Follow the instructions on the `#join-course` channel - Join the `#language-detection` channel Just make sure you comment here to indicate that you’ll be contributing to this project 🙂 ---
**Author:** ![ivanlau](https://avatars.discourse-cdn.com/v4/letter/i/b38774/32.png) [@ivanlau](https://discuss.huggingface.co/u/ivanlau)\ **Post date:** [November 16, 2021, 5:27am UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520/2 "2021-11-16T05:27:28Z")
@lewtun Hi, I would like to work on this project. 🙂 ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [November 16, 2021, 9:38am UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520/3 "2021-11-16T09:38:40Z")
Cool to hear @ivanlau ! I’ve created a Discord channel (see topic description) in case you and others want to use it 🙂 ---
**Author:** ![syri](https://avatars.discourse-cdn.com/v4/letter/s/c5a1d2/32.png) [@syri](https://discuss.huggingface.co/u/syri)\ **Post date:** [November 17, 2021, 10:00pm UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520/4 "2021-11-17T22:00:37Z")
@lewtun Hi, I’m interested in working on this project. Are we supposed to consider all the languages recognized by the `langid` library (which we’ll be using as baseline), or is it ok to consider fewer languages ? ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [November 17, 2021, 10:02pm UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520/5 "2021-11-17T22:02:02Z")
It’s totally fine for you to choose the scope for the project 🙂 I agree that working with a few languages is a great way to start! ---
**Author:** ![imimen](https://avatars.discourse-cdn.com/v4/letter/i/13edae/32.png) [@imimen](https://discuss.huggingface.co/u/imimen)\ **Post date:** [November 17, 2021, 10:09pm UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520/6 "2021-11-17T22:09:10Z")
Hi @lewtun ! I want to work on this project. ---
**Author:** ![syri](https://avatars.discourse-cdn.com/v4/letter/s/c5a1d2/32.png) [@syri](https://discuss.huggingface.co/u/syri)\ **Post date:** [November 17, 2021, 10:09pm UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520/7 "2021-11-17T22:09:30Z")
Alright thanks !! I’ll do so 🙂 ---
**Author:** ![papluca](https://avatars.discourse-cdn.com/v4/letter/p/45deac/32.png) [@papluca](https://discuss.huggingface.co/u/papluca)\ **Post date:** [November 17, 2021, 10:19pm UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520/8 "2021-11-17T22:19:24Z")
Hi, I’d be glad to join this project. Am I still in time? ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [November 17, 2021, 10:21pm UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520/9 "2021-11-17T22:21:02Z")
I think you’re the 4th person, so yes there’s still time and space on the team! ---
**Author:** ![hfawaz](https://avatars.discourse-cdn.com/v4/letter/h/ea666f/32.png) [@hfawaz](https://discuss.huggingface.co/u/hfawaz)\ **Post date:** [January 26, 2022, 10:28am UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520/10 "2022-01-26T10:28:20Z")
hi is this project still on ? ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [January 26, 2022, 10:30am UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520/11 "2022-01-26T10:30:12Z")
Hi @hfawaz, this community event ended last November. Having said that, you’re more than welcome to use the #course:course-event topics as inspiration to build NLP powered applications 🙂 ---
**Author:** ![hfawaz](https://avatars.discourse-cdn.com/v4/letter/h/ea666f/32.png) [@hfawaz](https://discuss.huggingface.co/u/hfawaz)\ **Post date:** [January 26, 2022, 10:30am UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520/12 "2022-01-26T10:30:56Z")
thanks, do you have any shared open ressources that you guys found following this course ? ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [January 26, 2022, 10:44am UTC](https://discuss.huggingface.co/t/build-a-language-detector/11520/13 "2022-01-26T10:44:27Z")
You might find the official tutorials in the `transformers` library to be helpful: [🤗 Transformers Notebooks](https://huggingface.co/docs/transformers/notebooks)