# Multilingual Visual Question Answering **URL:** \ **Category:** Flax/JAX Projects\ **Created:** [June 30, 2021, 6:50am UTC](https://discuss.huggingface.co/t/multilingual-visual-question-answering/7709 "2021-06-30T06:50:53Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![bharatR](https://avatars.discourse-cdn.com/v4/letter/b/b487fb/32.png) [@bharatR](https://discuss.huggingface.co/u/bharatR)\ **Post date:** [June 30, 2021, 6:50am UTC](https://discuss.huggingface.co/t/multilingual-visual-question-answering/7709/1 "2021-06-30T06:50:53Z")
# Multilingual Visual Question Answering We are currently planning to use ViT + mBert for two modalities namely images and text for Multilingual Visual Question Answering task ## Model Pre-trained models [ViT](https://huggingface.co/models?filter=vit) , mBERT , can be used for this task. ## 4. Datasets Currently we came across one dataset [WIT](https://github.com/google-research-datasets/wit) which is a large multimodal multilingual dataset , We can update the datasets if we encounter more suitable dataset for this task. ## 5. Training scripts Since this is a Seq2Seq model, the [run\_summarization\_flax.py](https://github.com/huggingface/transformers/tree/master/examples/flax/summarization) script can be used for training this model with some modifications. ## 6. (Optional) Challenges - How to combine the two modalities i.e. image and text, some of the ways to this is like [VisualBert](https://arxiv.org/abs/1908.03557) ,Here we can use ViT for images and use the embeddings from earlier stage for mBERT along with word embeddings. One other way for interaction of two modalities can be done through coattentional transformer layer like in [ViLBERT](https://arxiv.org/abs/1908.02265), so we need to decide which would be better suitable for our task. ## 7. (Optional) Desired project outcome Final Goal is to have an end to end model where we can perform Visual QA Task in multiple languages. We are also planning to benchmark the few-Shot/zero-shot performance on [VQA](https://visualqa.org/)/[VizWiz](https://vizwiz.org/tasks-and-datasets/vqa/)/[GQA](https://cs.stanford.edu/people/dorarad/gqa/about.html) ---
**Author:** ![gchhablani](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gchhablani/32/19723_2.png) [@gchhablani](https://discuss.huggingface.co/u/gchhablani)\ **Post date:** [June 30, 2021, 7:25am UTC](https://discuss.huggingface.co/t/multilingual-visual-question-answering/7709/2 "2021-06-30T07:25:56Z")
Count me in! ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [June 30, 2021, 11:55am UTC](https://discuss.huggingface.co/t/multilingual-visual-question-answering/7709/3 "2021-06-30T11:55:24Z")
Awesome let’s finalize this project 🙂 ---
**Author:** ![Ankit-Kumar-Saini](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@Ankit-Kumar-Saini](https://discuss.huggingface.co/u/Ankit-Kumar-Saini)\ **Post date:** [July 1, 2021, 12:18pm UTC](https://discuss.huggingface.co/t/multilingual-visual-question-answering/7709/4 "2021-07-01T12:18:57Z")
Great project. Looking to forward to working with you. ---
**Author:** ![tghosh93](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tghosh93/32/2130_2.png) [@tghosh93](https://discuss.huggingface.co/u/tghosh93)\ **Post date:** [July 1, 2021, 6:32pm UTC](https://discuss.huggingface.co/t/multilingual-visual-question-answering/7709/5 "2021-07-01T18:32:54Z")
I want to join this project. I hope I’m not late. ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [July 1, 2021, 11:52pm UTC](https://discuss.huggingface.co/t/multilingual-visual-question-answering/7709/6 "2021-07-01T23:52:43Z")
Added you guys! ---
**Author:** ![asharma85](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@asharma85](https://discuss.huggingface.co/u/asharma85)\ **Post date:** [July 2, 2021, 11:26am UTC](https://discuss.huggingface.co/t/multilingual-visual-question-answering/7709/7 "2021-07-02T11:26:38Z")
Hey all I am a beginner with Transformers and FLAX 😅 and want to get into Transformers via a project. Basically what I would like to take away is - knowledge about what it means to get a pre-trained model tuned to another task using HuggingFace’s infrastructure/pipeline and learn some FLAX basics I have sufficient background in CV and Deep Learning (Object detection.Semantic Seg etc via SSD, U-Net) in the autonomous driving space using PyTorch I do like this idea and would like to sign up. Do you think I can help out in some way? ---
**Author:** ![sarcasticvibes](https://avatars.discourse-cdn.com/v4/letter/s/4491bb/32.png) [@sarcasticvibes](https://discuss.huggingface.co/u/sarcasticvibes)\ **Post date:** [July 2, 2021, 5:47pm UTC](https://discuss.huggingface.co/t/multilingual-visual-question-answering/7709/8 "2021-07-02T17:47:10Z")
Hi guys, I would also like to be a part of this project. ---
**Author:** ![reichenbach](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/reichenbach/32/18911_2.png) [@reichenbach](https://discuss.huggingface.co/u/reichenbach)\ **Post date:** [July 2, 2021, 7:31pm UTC](https://discuss.huggingface.co/t/multilingual-visual-question-answering/7709/9 "2021-07-02T19:31:09Z")
Hi @patrickvonplaten, can you add me here in this Multilingual Visual QA ?