# Multilingual Image Captioning **URL:** \ **Category:** Flax/JAX Projects\ **Created:** [June 29, 2021, 6:02pm UTC](https://discuss.huggingface.co/t/multilingual-image-captioning/7671 "2021-06-29T18:02:39Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![bhavitvyamalik](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bhavitvyamalik/32/30215_2.png) [@bhavitvyamalik](https://discuss.huggingface.co/u/bhavitvyamalik)\ **Post date:** [June 29, 2021, 6:02pm UTC](https://discuss.huggingface.co/t/multilingual-image-captioning/7671/1 "2021-06-29T18:02:39Z")
We’re planning to use ViT encoder, mBART decoder and train them end-to-end for image captioning in different languages. ### Model Pre-trained [ViT](https://huggingface.co/models?filter=vit), [mBART](https://github.com/huggingface/transformers/pull/12236) (will be merged soon) can be leveraged for our task. ### Datasets Currently we’re thinking of using [WIT](https://github.com/google-research-datasets/wit). [COCO](https://cocodataset.org/#download) dataset can also be used but it has caption only in English language. Dataset can be changed later if we come across more suitable dataset for this task. ### Available training scripts Since this is a Seq2Seq model, the [run\_summarization\_flax.py](https://github.com/huggingface/transformers/tree/master/examples/flax/summarization) script can be used for training this model with some modifications. ### (Optional) Desired project outcome Our end use case would be to run this model on a video clip or movie and make it like an accessibility tool for visually challenged people. This can be showcased with a streamlit or gradio app. We’re also planning to benchmark its zero-shot performance on [VizWiz](https://vizwiz.org/workshops/2021-workshop/) ### (Optional) Challenges -Training encoder-decoder model end-to-end in FLAX/JAX will require some effort -Data Processing incase new datasets are available only in English ---
**Author:** ![gchhablani](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gchhablani/32/19723_2.png) [@gchhablani](https://discuss.huggingface.co/u/gchhablani)\ **Post date:** [June 29, 2021, 6:07pm UTC](https://discuss.huggingface.co/t/multilingual-image-captioning/7671/2 "2021-06-29T18:07:52Z")
I am in 😛 @abheesht @Vaibhavbrkn @nilshoehing have also shown interest in this sheet: [JAX/Flax Vision-and-Language Projects - Google Sheets](https://docs.google.com/spreadsheets/d/1tY-3mA4B6YxjpfgiM46GDq-Cgd-FyhfzzgcNFQ6Mn2k/edit?usp=sharing) Others are welcome to join! 🙂 ---
**Author:** ![nilshoehing](https://avatars.discourse-cdn.com/v4/letter/n/ce7236/32.png) [@nilshoehing](https://discuss.huggingface.co/u/nilshoehing)\ **Post date:** [June 29, 2021, 6:19pm UTC](https://discuss.huggingface.co/t/multilingual-image-captioning/7671/3 "2021-06-29T18:19:44Z")
Yes, I am also in! ---
**Author:** ![knilakshan20](https://avatars.discourse-cdn.com/v4/letter/k/f4b2a3/32.png) [@knilakshan20](https://discuss.huggingface.co/u/knilakshan20)\ **Post date:** [June 30, 2021, 3:45am UTC](https://discuss.huggingface.co/t/multilingual-image-captioning/7671/4 "2021-06-30T03:45:55Z")
I am also in 😉 ---
**Author:** ![Monda](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/monda/32/1293_2.png) [@Monda](https://discuss.huggingface.co/u/Monda)\ **Post date:** [June 30, 2021, 4:10am UTC](https://discuss.huggingface.co/t/multilingual-image-captioning/7671/5 "2021-06-30T04:10:35Z")
consider me on this 🙂 ---
**Author:** ![rays2pix](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@rays2pix](https://discuss.huggingface.co/u/rays2pix)\ **Post date:** [June 30, 2021, 4:48am UTC](https://discuss.huggingface.co/t/multilingual-image-captioning/7671/6 "2021-06-30T04:48:08Z")
I am interested in this task. ---
**Author:** ![rays2pix](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@rays2pix](https://discuss.huggingface.co/u/rays2pix)\ **Post date:** [June 30, 2021, 4:50am UTC](https://discuss.huggingface.co/t/multilingual-image-captioning/7671/7 "2021-06-30T04:50:27Z")
We can also use this dataset [GitHub - bharathichezhiyan/multimodalmachinetranslation-Tamil: Multilingual Multimodal Machine Translation for Dravidian Languages utilizing Phonetic Transcription](https://github.com/bharathichezhiyan/multimodalmachinetranslation-Tamil) ---
**Author:** ![abheesht](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/abheesht/32/1966_2.png) [@abheesht](https://discuss.huggingface.co/u/abheesht)\ **Post date:** [June 30, 2021, 8:45am UTC](https://discuss.huggingface.co/t/multilingual-image-captioning/7671/8 "2021-06-30T08:45:25Z")
Yep, I’m in, @bhavitvyamalik, @gchhablani 🙂 ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [June 30, 2021, 10:13am UTC](https://discuss.huggingface.co/t/multilingual-image-captioning/7671/9 "2021-06-30T10:13:19Z")
Great Idea! One important thing is to define the scope of the project, specifically which and how many languages to use. We should decide this based on the time and compute constraint so that the project can be finished in time 🙂 It’s also possible to use other multilingual models like mBERT of XLM-ROBERTa, but as those are encoder-only models we will need to add a cross-attention layer in it. I’m officially defining this project! Added you the team @bhavitvyamalik , @gchhablani , @nilshoehing , @knilakshan20 , @Monda , @rays2pix , @abheesht , @Vaibhavbrkn Let me know if you have any comments 🙂 ---
**Author:** ![asharma85](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@asharma85](https://discuss.huggingface.co/u/asharma85)\ **Post date:** [July 2, 2021, 11:13am UTC](https://discuss.huggingface.co/t/multilingual-image-captioning/7671/10 "2021-07-02T11:13:05Z")
Hey all I am a beginner with Transformers and FLAX 😅 and want to get into Transformers via a project. Basically I would like to be able to work on getting a minimal training pipeline built and a model trained using HuggingFace’s infra I have sufficient background in CV and Deep Learning (Object detection.Semantic Seg etc via SSD, U-Net) in the autonomous driving space I do like this idea and would like to sign up. Do you think I can help out in some way? ---
**Author:** ![tejas2811](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tejas2811/32/2204_2.png) [@tejas2811](https://discuss.huggingface.co/u/tejas2811)\ **Post date:** [July 6, 2021, 12:01pm UTC](https://discuss.huggingface.co/t/multilingual-image-captioning/7671/11 "2021-07-06T12:01:53Z")
I would like to work on this Project too.