# About extracting text information as well as relevant images from document likes pdf doc etc

**URL:** <https://discuss.huggingface.co/t/about-extracting-text-information-as-well-as-relevant-images-from-document-likes-pdf-doc-etc/75874>\
**Category:** Community Calls\
**Created:** [March 4, 2024, 5:14pm UTC](https://discuss.huggingface.co/t/about-extracting-text-information-as-well-as-relevant-images-from-document-likes-pdf-doc-etc/75874 "2024-03-04T17:14:24Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Adarsh123](https://avatars.discourse-cdn.com/v4/letter/a/f4b2a3/32.png) [@Adarsh123](https://discuss.huggingface.co/u/Adarsh123)\
**Post date:** [March 4, 2024, 5:14pm UTC](https://discuss.huggingface.co/t/about-extracting-text-information-as-well-as-relevant-images-from-document-likes-pdf-doc-etc/75874/1 "2024-03-04T17:14:24Z")

</div>

Use case is to extract the relevant text information along with images available in the file using generative ai, When any prompt is given then relevant text information and image should display as response.

Kindly help by providing some ideas, links or techniques.  
Thank you.

---

<div class="post-metadata">

**Author:** ![FredS](https://avatars.discourse-cdn.com/v4/letter/f/ba9def/32.png) [@FredS](https://discuss.huggingface.co/u/FredS)\
**Post date:** [May 5, 2024, 9:56am UTC](https://discuss.huggingface.co/t/about-extracting-text-information-as-well-as-relevant-images-from-document-likes-pdf-doc-etc/75874/2 "2024-05-05T09:56:52Z")

</div>

Search for MultiModal RAG

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [May 6, 2024, 6:58am UTC](https://discuss.huggingface.co/t/about-extracting-text-information-as-well-as-relevant-images-from-document-likes-pdf-doc-etc/75874/3 "2024-05-06T06:58:37Z")

</div>

Yes indeed this is a multimodal RAG use case.

Something that often works well is summarizing the images using an image-text-to-text model (could be an open-source one like LlaVa, LlaVa-NeXT, Idefics2, etc. which are best among these at the time of writing, or a closed-source one like GPT-4V, Gemini).

Then you can embed the image summarizations just like regular text, store the embeddings in a vector database and perform regular RAG.

---

<div class="post-metadata">

**Author:** ![hugolb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hugolb/32/29912_2.png) [@hugolb](https://discuss.huggingface.co/u/hugolb)\
**Post date:** [July 27, 2024, 12:40pm UTC](https://discuss.huggingface.co/t/about-extracting-text-information-as-well-as-relevant-images-from-document-likes-pdf-doc-etc/75874/4 "2024-07-27T12:40:09Z")

</div>

checkout this cookbook, seems very easy to do a multimodal RAG with this library:  
[https://github.com/ntropy-ai/ntropy/blob/main/examples/sp500-report/main.ipynb](https://github.com/ntropy-ai/ntropy/blob/main/examples/sp500-report/main.ipynb)