# Image Captioning - ViT + BERT with WIT

**URL:** <https://discuss.huggingface.co/t/image-captioning-vit-bert-with-wit/8131>\
**Category:** Flax/JAX Projects\
**Created:** [July 7, 2021, 9:09am UTC](https://discuss.huggingface.co/t/image-captioning-vit-bert-with-wit/8131 "2021-07-07T09:09:28Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dmatos2012](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dmatos2012/32/1878_2.png) [@dmatos2012](https://discuss.huggingface.co/u/dmatos2012)\
**Post date:** [July 7, 2021, 9:09am UTC](https://discuss.huggingface.co/t/image-captioning-vit-bert-with-wit/8131/1 "2021-07-07T09:09:28Z")

</div>

Hi,

So am fairly new at NLP, so im just getting used to HF and NLP in general. My project is at [img-caption in spanish](https://discuss.huggingface.co/t/image-captioning-for-spanish-with-pre-trained-vision-and-text-model/7084). Thus following questions,

1. IT looks like [wit](https://github.com/google-research-datasets/wit) dataset isnt on HF library, so downloaded it on the TPU. But files are on tsv and the HF parsers I found support csv, py dicts, json etcs. What should be the fastest way here to parse correctly? any HF library for TSVs?

2. [Valhalla](https://huggingface.co/valhalla) my proj lead suggested using [run\_summarization\_flax.py](https://github.com/huggingface/transformers/tree/master/examples/flax/summarization) script, but I am a bit confused. How would I go about specifying ViT as encoder; and how would I go about adding my cross-attn layer to BERT and specifying it to that script ? I see only one `model_dir` and `tokenizer_name` which I am not sure for my use case.

I just want to start with pre-trained networsk and once it works, go with fine tuning etc. Thanks

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [July 7, 2021, 9:22am UTC](https://discuss.huggingface.co/t/image-captioning-vit-bert-with-wit/8131/2 "2021-07-07T09:22:18Z")

</div>

Hi there,

Regrading the dataset:  
Yes, the dataset is not available in `datasets` lib. You will need to download the tsv and the prepare the dataset by downloading the image. The tsv file for wit contains the image URLs and other metadata.  
This script might [help](https://github.com/borisdayma/dalle-mini/blob/main/data/CC3M_downloader.py). It’s for downloading conceptual captions data, but you could re-purpose it to download WIT.

Regarding model:  
There is no off-the-shelf model for this in `transformers` (yet!). What we need here is a Seq2Seq model. Where that image encoder is a pre-trained vision model like `ViT` or `CLIP`'s vision model and the decoder is any pre-trained text model (BERT/ROBERTa).

To do this we will need to modify the BERT/ROBERTa model and add a cross-attention layer in it.  
The encoder will the encode images, the decoder will take the `decoder_input_ids` and the `encoder_hidden_states`.

This should be similar to encoder-decoder model in pytorch [transformers/modeling\_encoder\_decoder.py at master · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/master/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py).

@gchhablani is also working on a similar project and might have the modeling code ready 🙂

Let me know if you have any more question.

---

<div class="post-metadata">

**Author:** ![johnrodriguez190380](https://avatars.discourse-cdn.com/v4/letter/j/ce7236/32.png) [@johnrodriguez190380](https://discuss.huggingface.co/u/johnrodriguez190380)\
**Post date:** [October 21, 2021, 7:19am UTC](https://discuss.huggingface.co/t/image-captioning-vit-bert-with-wit/8131/4 "2021-10-21T07:19:24Z")

</div>

Any progress on this topic?