# Vision-Language Project Ideas **URL:** \ **Category:** Flax/JAX Projects\ **Created:** [June 25, 2021, 12:44pm UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356 "2021-06-25T12:44:16Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![gchhablani](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gchhablani/32/19723_2.png) [@gchhablani](https://discuss.huggingface.co/u/gchhablani)\ **Post date:** [June 25, 2021, 12:44pm UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/1 "2021-06-25T12:44:16Z")
After some brainstorming, we came with the following project ideas. Would love some feedback/opinion on the same. - ViT + mBART - Multilingual Image Captioning (WIT pre-train) - ViT + mBERT - Multilingual Visual Question Answering (WIT/COCO pre-train, test on VQA/GQA) - Use CLIP/VQGAN for Image Synthesis. A project has been proposed GIF generation. We can train on a different dataset/domain. Possible modifications of CLIP/VQGAN: - Multilingual CLIP model for image/sentence matching + Image Generation using VQGAN for this dataset. - FashionCLIP- Train a model for Fashion image-text matching and VQGAN, trained to generate dress/shirt/glasses based on description. Dataset is hard to find, I guess. - Scene generation - CLIP + VQGAN, trained on text-scene dataset for scene (movie/landscape) generation. Alternatively, can do similar things for video+text, following the example of VideoBERT and other transformers, maybe? Please comment here if you are interested in collaboration. We are a team of 5 as of now (timezone GMT+5:30). Would also love any suggestions to improve these ideas. Team: - @abheesht - @bhavitvyamalik - @Divyadixit - @gchhablani - @harsh68 Thanks, Gunjan ---
**Author:** ![Vaibhavbrkn](https://avatars.discourse-cdn.com/v4/letter/v/74df32/32.png) [@Vaibhavbrkn](https://discuss.huggingface.co/u/Vaibhavbrkn)\ **Post date:** [June 25, 2021, 1:26pm UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/2 "2021-06-25T13:26:15Z")
@gchhablani very interesting ideas, I have experience in CV and NLP and would like to contribute in this project. ---
**Author:** ![knilakshan20](https://avatars.discourse-cdn.com/v4/letter/k/f4b2a3/32.png) [@knilakshan20](https://discuss.huggingface.co/u/knilakshan20)\ **Post date:** [June 26, 2021, 8:25am UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/3 "2021-06-26T08:25:58Z")
@gchhablani I am interested in VQA project, I have domain experience in both vision and NLP(intermediate). What kind of expertise you wish to join the iteam if it is not completed? Thanks ---
**Author:** ![gchhablani](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gchhablani/32/19723_2.png) [@gchhablani](https://discuss.huggingface.co/u/gchhablani)\ **Post date:** [June 26, 2021, 8:48am UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/4 "2021-06-26T08:48:48Z")
Hi @knilakshan20 We will choose one of these topics after discussing. If you’re up for other projects too, it would be great to have you with us. Once we have finalized a project, we will share another post on the forum. Maybe then you can decide, otherwise. ---
**Author:** ![gchhablani](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gchhablani/32/19723_2.png) [@gchhablani](https://discuss.huggingface.co/u/gchhablani)\ **Post date:** [June 26, 2021, 11:16am UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/6 "2021-06-26T11:16:39Z")
Hi @Sasikanth Is there a specific one you are interested in? Which ones do you think are nice? And what can be improved? Any other suggestions that you have? We will pick one of these and add another post. If there are enough people who reply here, we can pick two projects from here and work on them separately, wdyt? ---
**Author:** ![lkhphuc](https://avatars.discourse-cdn.com/v4/letter/l/2bfe46/32.png) [@lkhphuc](https://discuss.huggingface.co/u/lkhphuc)\ **Post date:** [June 26, 2021, 5:07pm UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/8 "2021-06-26T17:07:30Z")
Hi, I’m interested in these projects, especially replicating VideoBERT as that is my research topic. Can I join in with you guys? ---
**Author:** ![bharatR](https://avatars.discourse-cdn.com/v4/letter/b/b487fb/32.png) [@bharatR](https://discuss.huggingface.co/u/bharatR)\ **Post date:** [June 27, 2021, 1:23pm UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/9 "2021-06-27T13:23:38Z")
Hi @gchhablani I am interested in the ViT + mBERT - Multilingual Visual Question Answering project, i have read a paper (a bit old ,published in 2019) related to this(their model is not limited to VQA task only,but we can finetune it), you can look at here: [GitHub - facebookresearch/vilbert-multi-task: Multi Task Vision and Language](https://github.com/facebookresearch/vilbert-multi-task) ---
**Author:** ![knilakshan20](https://avatars.discourse-cdn.com/v4/letter/k/f4b2a3/32.png) [@knilakshan20](https://discuss.huggingface.co/u/knilakshan20)\ **Post date:** [June 27, 2021, 3:28pm UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/10 "2021-06-27T15:28:47Z")
I am comfortable with image captioning too. I have some experience intense projects. I am interested to learn on videbert model. ---
**Author:** ![Dimitre](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dimitre/32/2058_2.png) [@Dimitre](https://discuss.huggingface.co/u/Dimitre)\ **Post date:** [June 27, 2021, 3:47pm UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/11 "2021-06-27T15:47:17Z")
Hey there I am also very interested in multi-modal models, I liked a lot the first two project ideas. - ViT + mBART - Multilingual Image Captioning (WIT pre-train) - ViT + mBERT - Multilingual Visual Question Answering (WIT/COCO pre-train, test on VQA/GQA) I would love to joing the team, if you wanna know a little more about my background check out my [GitHub](https://github.com/dimitreOliveira). ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [June 28, 2021, 4:42pm UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/14 "2021-06-28T16:42:41Z")
Hey guys these are great ideas! Would be nice if you could open different threads for different projects and comment there if you want to be a part of that project, this is so that we can keep track of teams and projects. Would be nice to do this before Wednesday 🙂 ---
**Author:** ![gchhablani](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gchhablani/32/19723_2.png) [@gchhablani](https://discuss.huggingface.co/u/gchhablani)\ **Post date:** [June 28, 2021, 5:16pm UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/15 "2021-06-28T17:16:15Z")
Hi everyone! Thanks for showing interest. Everyone has different interests/expertise which could be very useful. However, since we only have one week, we can only pursue few ideas. I am thinking we can pick two of these. I am maintaining this tiny sheet based on which we might be able to divide ourselves into two teams: > **[JAX/Flax Vision-and-Language Projects](https://docs.google.com/spreadsheets/d/1tY-3mA4B6YxjpfgiM46GDq-Cgd-FyhfzzgcNFQ6Mn2k/edit?usp=sharing)** > > Sheet1 > > Name,HF user,GitHub User,Preference 1,Preference 2,Preference 3,Other Preferences,Timezone,Availability per Day from July 1-14,Expertise/Past Projects,Ideas to Improve Projects,Concerns > Gunjan... Please fill it by tomorrow so the teams and meet and decide on specifics of the project by Wednesday and post them here. Right now we are a total of 14 people, so I am thinking 7/7 will be ideal, unless some of you aren’t interested in this anymore and have found something better 🙂 EDIT: Please also join the Discord group here: [Flax-HuggingFace-Community-Week](https://discord.gg/wXKZHHmq) ---
**Author:** ![Tanweer](https://avatars.discourse-cdn.com/v4/letter/t/838e76/32.png) [@Tanweer](https://discuss.huggingface.co/u/Tanweer)\ **Post date:** [June 28, 2021, 6:09pm UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/16 "2021-06-28T18:09:23Z")
I find the ideas very interesting and will like to contribute to both the projects, ViT + mBERT - Multilingual Visual Question Answering (WIT/COCO pre-train, test on VQA/GQA) or CLIP/VQGAN for Image Synthesis. I am really interested in collaborating and contributing in this project. ---
**Author:** ![nilshoehing](https://avatars.discourse-cdn.com/v4/letter/n/ce7236/32.png) [@nilshoehing](https://discuss.huggingface.co/u/nilshoehing)\ **Post date:** [June 28, 2021, 9:44pm UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/17 "2021-06-28T21:44:56Z")
Hi, I would also like to join the image captioning or the VQA Topic, if it is still possible. I have some experience with nlp and tensorflow and I am located in GMT+2 ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [June 30, 2021, 9:15am UTC](https://discuss.huggingface.co/t/vision-language-project-ideas/7356/18 "2021-06-30T09:15:01Z")
Hi @gchhablani Please let me know when you have formed the teams and projects so I could add you to the [sheet](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit?usp=sharing).