# How do I use Text-Image to Text models with Huggingface Inference?

**URL:** <https://discuss.huggingface.co/t/how-do-i-use-text-image-to-text-models-with-huggingface-inference/111514>\
**Category:** Beginners\
**Created:** [October 12, 2024, 7:14am UTC](https://discuss.huggingface.co/t/how-do-i-use-text-image-to-text-models-with-huggingface-inference/111514 "2024-10-12T07:14:04Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![perceptron-743](https://avatars.discourse-cdn.com/v4/letter/p/85e7bf/32.png) [@perceptron-743](https://discuss.huggingface.co/u/perceptron-743)\
**Post date:** [October 12, 2024, 7:14am UTC](https://discuss.huggingface.co/t/how-do-i-use-text-image-to-text-models-with-huggingface-inference/111514/1 "2024-10-12T07:14:04Z")

</div>

Hi all, I want to use [phi-3.5-text-img-text](https://huggingface.co/microsoft/Phi-3.5-vision-instruct) for a multi modal task which takes the image of a page and converts the contents of the page into html tags and if there is an image found on the page, it will convert that image into text and add ‘illustration’ tag beside it.

How do I use the huggingface inference api for [phi3.5instruct](https://api-inference.huggingface.co/models/microsoft/Phi-3.5-vision-instruct) so that it generates the output.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 12, 2024, 1:24pm UTC](https://discuss.huggingface.co/t/how-do-i-use-text-image-to-text-models-with-huggingface-inference/111514/2 "2024-10-12T13:24:28Z")

</div>

It’s in the manual, but it’s a newly implemented pipeline, so I don’t know if it really works.

> **[What is Image-Text-to-Text? - Hugging Face](https://huggingface.co/tasks/image-text-to-text)**
>
> Learn about Image-Text-to-Text using Machine Learning

> **[Image-text-to-text](https://huggingface.co/docs/transformers/tasks/image_text_to_text)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![not-lain](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/not-lain/32/59092_2.png) [@not-lain](https://discuss.huggingface.co/u/not-lain)\
**Post date:** [October 12, 2024, 3:12pm UTC](https://discuss.huggingface.co/t/how-do-i-use-text-image-to-text-models-with-huggingface-inference/111514/3 "2024-10-12T15:12:36Z")

</div>

Hi @perceptron-743  
the model `microsoft/Phi-3.5-vision-instruct` contains `custom-code`.  
By default, inference api is disabled for such models, this is done for security reasons since you can’t know what code the model is running in the back.  
another way to see if the model has an inference api disabled is to go to the model page and check on the right

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/e/f/ef60be6eb1df642a322e873d123bace58bc89312.png)  
if the logo has a broken ligthening logo it means the inference api is disabled for that model.  
To filter the hub by models that have inference api enabled you can checkout the warm and cold ones as shown in the picture below  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/d/4/d43b8bd59baca6618326b9c7cd083ac5a096ba26.jpeg)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 12, 2024, 10:56pm UTC](https://discuss.huggingface.co/t/how-do-i-use-text-image-to-text-models-with-huggingface-inference/111514/4 "2024-10-12T22:56:08Z")

</div>

You’re right, but isn’t the server-side behavior a bit buggy?  
When Inference is turned off, the input field itself on the GUI page doesn’t appear, which is normal behavior for other inferences.  
Moreover, the author has not explicitly turned off Inference, and he appears to have done the user-enabled settings to turn it on.

```auto
---
license: mit
license_link: https://huggingface.co/microsoft/Phi-3.5-vision-instruct/resolve/main/LICENSE
language:
- multilingual
pipeline_tag: image-text-to-text
tags:
- nlp
- code
- vision
inference:
  parameters:
    temperature: 0.7
widget:
- messages:
  - role: user
    content: <|image_1|>Can you describe what you see in the image?
library_name: transformers
---

```