# Multimodal LLM with Image and Text sequentially in its prompt

**URL:** <https://discuss.huggingface.co/t/multimodal-llm-with-image-and-text-sequentially-in-its-prompt/65210>\
**Category:** 🤗Transformers\
**Created:** [December 10, 2023, 6:07am UTC](https://discuss.huggingface.co/t/multimodal-llm-with-image-and-text-sequentially-in-its-prompt/65210 "2023-12-10T06:07:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![purnasai](https://avatars.discourse-cdn.com/v4/letter/p/8e7dd6/32.png) [@purnasai](https://discuss.huggingface.co/u/purnasai)\
**Post date:** [December 10, 2023, 6:07am UTC](https://discuss.huggingface.co/t/multimodal-llm-with-image-and-text-sequentially-in-its-prompt/65210/1 "2023-12-10T06:07:40Z")

</div>

I am trying to find a Single LLM Model that can Accept both images and text in the prompt.

Models like LLava-13B, FuYu-8B expects Image+text in Every prompt. check [here](https://huggingface.co/docs/transformers/main/model_doc/llava). prompt looks as

“\< image \>\nUSER: What’s the content of the image?\nASSISTANT:”

Models like Laama, mistral expects only text in Every prompt.

Building 2 individual modules in the pipeline, might lack context while user inputs questions and images in chat. So I am looking for a Single LM.

In my exploration I have found [Next-GPT](https://next-gpt.github.io/), can have a chat with using both image sometimes and text only sometimes. I want to use a single LLM. But I couldn’t find its integration in HF.

Any help I could use from the forum would be so much helpful. Tagging @sgugger @nielsr @sayakpaul for quick response.

Thank you so much in advance

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [December 10, 2023, 5:35pm UTC](https://discuss.huggingface.co/t/multimodal-llm-with-image-and-text-sequentially-in-its-prompt/65210/2 "2023-12-10T17:35:33Z")

</div>

Hi,

One such model is [CogVLM](https://huggingface.co/THUDM/cogvlm-chat-hf), which can be used both as a standalone large language model (text-only), accepting images as additional input.

---

<div class="post-metadata">

**Author:** ![purnasai](https://avatars.discourse-cdn.com/v4/letter/p/8e7dd6/32.png) [@purnasai](https://discuss.huggingface.co/u/purnasai)\
**Post date:** [January 1, 2024, 11:36am UTC](https://discuss.huggingface.co/t/multimodal-llm-with-image-and-text-sequentially-in-its-prompt/65210/3 "2024-01-01T11:36:32Z")

</div>

Thank you @nielsr. I tried & failed to load CogVLM in colab. So I used Qwen-7b & has loaded it in colab. I used in my application.

Thank you