# Multimodal training **URL:** \ **Category:** 🤗Transformers\ **Created:** [March 20, 2025, 8:40pm UTC](https://discuss.huggingface.co/t/multimodal-training/146698 "2025-03-20T20:40:55Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![celalp](https://avatars.discourse-cdn.com/v4/letter/c/8edcca/32.png) [@celalp](https://discuss.huggingface.co/u/celalp)\ **Post date:** [March 20, 2025, 8:40pm UTC](https://discuss.huggingface.co/t/multimodal-training/146698/1 "2025-03-20T20:40:55Z")
Hi, I have a dataset that consists of images, their captions (they are scientific figures) and some excerpts from the paper main text that references the figure. The goal of this is to for a given figure and its caption, can we understand the figure (the text in the paper). This is different from an image captioning problem but more of a reasoning problem. I would appreciate any pointers on how to train on image-text pairs as input and text as output. In this instance the figure captions are quite important because many figures look alike even within a paper and the figure caption is important to differentiate between them. Thanks for all the suggestions in advance. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [March 21, 2025, 7:20am UTC](https://discuss.huggingface.co/t/multimodal-training/146698/2 "2025-03-21T07:20:52Z")
In your case, I think you would want to combine VLM and LLM to perform VQA-like tasks. You could train each lightweight model separately and then combine them, or some high-performance VLMs already have quite LLM-like capabilities. However, I think a model like LLaVA, which is a combination of VLM and LLM, would be more suitable. > **[LLaVA-o1: Let Vision Language Models Reason Step-by-Step](https://huggingface.co/blog/mikelabs/llava-o1-let-vision-language-models-reason)** > > A Blog post by Mike Young on Hugging Face > **[ColPali: Efficient Document Retrieval with Vision Language Models 👀](https://huggingface.co/blog/manu/colpali)** > > A Blog post by Manuel Faysse on Hugging Face ### VLMs > **[CohereLabs/aya-vision-8b · Hugging Face](https://huggingface.co/CohereLabs/aya-vision-8b)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[Qwen/Qwen2.5-VL-7B-Instruct · Hugging Face](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[Announcing Pixtral 12B | Mistral AI](https://mistral.ai/news/pixtral-12b)** > > Pixtral 12B - the first-ever multimodal Mistral model. Apache 2.0. ### Other approaches by [Hugging Chat](https://huggingface.co/chat/) * * * Based on the sources provided, here are effective approaches and models for training on image-text pairs to understand scientific figures and generate reasoned text outputs: * * * ### 1. **Contrastive Learning with Captioning Models** - **Model** : CoCa (Contrastive Captioner) [1] - **Model** : Mistral 7B [3] * * * ### 2. **Explicit Image Caption Reasoning (ECR)** - **Model** : ECRMM (Explicit Caption Reasoning Multimodal Model) [4] - ECR employs inference chaining to analyze images deeply and generate detailed captions. It is particularly effective for complex scenes and fine-grained information. - **Key Features** : - Focuses on reasoning and semantic parsing for accurate and detailed descriptions. - Fine-tuned on datasets like ICICD, which includes images, captions, and textual context. - **Use Case** : Suitable for your dataset, as it emphasizes understanding the relationships between images, captions, and textual context. * * * ### 3. **Contrastive Learning and Multi-Modal Training** - **Approach** : Contrastive learning [2][4] - **Model** : Multi-Modal Transformers [2] * * * ### Recommendations - **Start with CoCa** for its strong performance in image-text alignment and caption generation. - Fine-tune Mistral 7B or ECRMM on your dataset to leverage advanced scene understanding and reasoning capabilities. - Use contrastive learning to align images with their captions, especially when figures are visually similar. * * * ### References - [1] Learn CoCa: Image-Text Foundation Models with Contrastive Captioners [Source] - [2] Multimodal training - 🤗Transformers - Hugging Face Forums [Source] - [3] Image Captioning with Mistral 7B LLM: A Hands-on Guide [Source] - [4] Explicit Image Caption Reasoning (ECR) [Source] ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [March 21, 2025, 7:26am UTC](https://discuss.huggingface.co/t/multimodal-training/146698/3 "2025-03-21T07:26:04Z")
### Training Tips > **[Multimodal Tasks and Models - Hugging Face Community Computer Vision Course](https://huggingface.co/learn/computer-vision-course/en/unit4/multimodal-models/tasks-models-part1)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[Fine-Tuning a Vision Language Model (Qwen2-VL-7B) with the Hugging Face...](https://huggingface.co/learn/cookbook/fine_tuning_vlm_trl)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[Accelerating Document AI](https://huggingface.co/blog/document-ai)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[Vision Fine-tuning | Unsloth Documentation](https://docs.unsloth.ai/basics/vision-fine-tuning)** > > Learn how to fine-tune vision/multimodal LLMs with Unsloth ---
**Author:** ![celalp](https://avatars.discourse-cdn.com/v4/letter/c/8edcca/32.png) [@celalp](https://discuss.huggingface.co/u/celalp)\ **Post date:** [March 21, 2025, 3:21pm UTC](https://discuss.huggingface.co/t/multimodal-training/146698/4 "2025-03-21T15:21:23Z")
Oh wow thank @John6666 for the detailed answers. I will check the models and references out. ---
**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\ **Post date:** [March 25, 2025, 7:38pm UTC](https://discuss.huggingface.co/t/multimodal-training/146698/5 "2025-03-25T19:38:51Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.