# Image captioning for Spanish with pre-trained vision and text model **URL:** \ **Category:** Flax/JAX Projects\ **Created:** [June 23, 2021, 10:33am UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084 "2021-06-23T10:33:49Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [June 23, 2021, 10:33am UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/1 "2021-06-23T10:33:49Z")
## Image captioning for Spanish with pre-trained vision and text model For this project, a pre-trained image model like `ViT` can be used as an encoder, and a pre-trained text model like BERT and/or GPT2 can be used as a decoder. ### Model Pre-trained [ViT](https://huggingface.co/models?filter=vit), [BERT](https://huggingface.co/bert-base-multilingual-cased) models can be found on the model hub. We could use multi-lingual BERT/ROERTa models for the Spanish language. ### Datasets The [WIT](https://github.com/google-research-datasets/wit) dataset can be used for this task. It has almost over 1M image-text pairs for Spanish. ### Available training scripts As this will be a Seq2Seq model, the [run\_summarization\_flax.py](https://github.com/huggingface/transformers/tree/master/examples/flax/summarization) script can be used for training this model with some modifications. ### (Optional) Desired project outcome The desired outcome is to see if pre-trained vision and text models can be leveraged for image captioning and also train captioning models in for the Spanish language. This can be showcased with a streamlit or gradio app. ### (Optional) Challenges This model will require some modifications to the existing text models. Specifically, as this will be a seq2seq model, we’ll need to add a randomly initialized cross-attention layer in BERT or GPT2 to use it as a decoder in the encoder-decoder setting. ### (Optional) Links to read upon This keras example presents an excellent example of how image encoder and transformer can be used for image captioning. > **[Keras documentation: Image Captioning](https://keras.io/examples/vision/image_captioning/)** > > Keras documentation: Image Captioning ---
**Author:** ![dmatos2012](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dmatos2012/32/1878_2.png) [@dmatos2012](https://discuss.huggingface.co/u/dmatos2012)\ **Post date:** [June 23, 2021, 3:48pm UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/2 "2021-06-23T15:48:56Z")
Hi, I would like to join this project. I am not experienced enough with NLP to create a project on my own. But I do have experience with Computer Vision overall, and would like to try and see how well spanish text is generated for the image-captioning. I am David and I am in the Amsterdam timezone(CET). I have some experience with other frameworks, and training/evaluating object detection networks, but not with NLP. I can contribute by putting in my previous knowledge and discipline towards achieving this goal 🙂 ---
**Author:** ![mrm8488](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mrm8488/32/5836_2.png) [@mrm8488](https://discuss.huggingface.co/u/mrm8488)\ **Post date:** [June 23, 2021, 3:54pm UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/3 "2021-06-23T15:54:43Z")
So you want to create a CLIP like (dual encoder) model? @valhalla ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [June 23, 2021, 5:13pm UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/4 "2021-06-23T17:13:55Z")
Hey @dmatos2012 , don’t worry about experience. We always try to make things easier for everyone and we have a super cool speaker lineup for getting familiar with JAX/Flax/Transformers. And we will try to answer all questions:) @mrm8488 For image captioning it’ll be more like an encoder-decoder model. The encoder will be an image model and the decoder can be any transformer model with cross-attention which will take `hidden_states` from image model and will generate text auto-regressively ---
**Author:** ![dmatos2012](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dmatos2012/32/1878_2.png) [@dmatos2012](https://discuss.huggingface.co/u/dmatos2012)\ **Post date:** [June 23, 2021, 9:29pm UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/5 "2021-06-23T21:29:57Z")
Awesome @valhalla! thanks ---
**Author:** ![Dimitre](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dimitre/32/2058_2.png) [@Dimitre](https://discuss.huggingface.co/u/Dimitre)\ **Post date:** [June 27, 2021, 12:44am UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/6 "2021-06-27T00:44:24Z")
Hey there, this is very interesting, I have some experience with NLP and computer vision, and always wanted to get more experience with multi-modal models (text + vision), also since I saw the WIT dataset for the first time, I wanted to use it for some project, this seems a good opportunity. If you want to know a little more about my background, check out my [GitHub](https://github.com/dimitreOliveira). ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [June 29, 2021, 3:24pm UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/7 "2021-06-29T15:24:46Z")
Great, let’s define the project! @mrm8488 let me know if you want to be added here as well 🙂 ---
**Author:** ![srisweet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/srisweet/32/1829_2.png) [@srisweet](https://discuss.huggingface.co/u/srisweet)\ **Post date:** [June 30, 2021, 3:26am UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/8 "2021-06-30T03:26:38Z")
Hello @valhalla @patrickvonplaten , Hope you are doing well… I am interested in this project as well… So much to learn from doing this project…It is similar to CLIP. But in Clip, we use encoders for both image & text… In this project, we are gonna use encoder for image & decoder for text. Also, I have a suggestion that we could create a separate thread indexed with all JAX/FLAX learning resources together in 1 place…It will be useful for all of us to learn… Cheers, Sri Lakshmi ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [June 30, 2021, 8:00am UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/9 "2021-06-30T08:00:56Z")
Hey @srisweet ! Added you the team 🙂 Great idea, feel free to create a thread to post JAX/Flax resources! ---
**Author:** ![dmatos2012](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dmatos2012/32/1878_2.png) [@dmatos2012](https://discuss.huggingface.co/u/dmatos2012)\ **Post date:** [June 30, 2021, 8:41am UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/10 "2021-06-30T08:41:55Z")
Hi @valhalla . could you add me as well please? I replied at the top of the post :). Thank you! ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [June 30, 2021, 8:58am UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/11 "2021-06-30T08:58:21Z")
hi @dmatos2012 you are already part of the team 🙂 Here’s the [sheet](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit?usp=sharing) ---
**Author:** ![dmatos2012](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dmatos2012/32/1878_2.png) [@dmatos2012](https://discuss.huggingface.co/u/dmatos2012)\ **Post date:** [June 30, 2021, 9:00am UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/12 "2021-06-30T09:00:21Z")
oh @valhalla apologies, did not see that. Sorry! ---
**Author:** ![srisweet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/srisweet/32/1829_2.png) [@srisweet](https://discuss.huggingface.co/u/srisweet)\ **Post date:** [June 30, 2021, 9:54am UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/13 "2021-06-30T09:54:10Z")
Hi @valhalla , Just created this thread with JAX/Flax resources… > [@FLAX/JAX Learning Resources](https://discuss.huggingface.co/t/flax-jax-learning-resources/7725): > > Hello all, I have tried my best to organize FLAX/JAX learning resources. I hope it will be useful for all of us. If you have more learning resources, drop a comment & I will add them here… [JAX](https://github.com/google/jax) is a numerical computation library that exposes a NumPy-like API with tracing capabilities. Flax is the most widely used JAX library, with [129 dependent projects](https://github.com/google/flax/network/dependents?package_id=UGFja2FnZS01MjEyMjA2MA%3D%3D) as of May 2021. [Flax](https://github.com/google/flax) builds on top of JAX with an ergonomic module abstraction using Python dataclasses that leads to concise and explicit c… ---
**Author:** ![bhavitvyamalik](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bhavitvyamalik/32/30215_2.png) [@bhavitvyamalik](https://discuss.huggingface.co/u/bhavitvyamalik)\ **Post date:** [July 19, 2021, 8:13am UTC](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084/15 "2021-07-19T08:13:49Z")
Hi Suraj, Me and @gchhablani are working on spanish image captioning currently with CLIP + Marian