# Streaming output text when deploying on Sagemaker

**URL:** <https://discuss.huggingface.co/t/streaming-output-text-when-deploying-on-sagemaker/39611>\
**Category:** Amazon SageMaker\
**Created:** [May 12, 2023, 7:09am UTC](https://discuss.huggingface.co/t/streaming-output-text-when-deploying-on-sagemaker/39611 "2023-05-12T07:09:49Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ludivine](https://avatars.discourse-cdn.com/v4/letter/l/ecae2f/32.png) [@Ludivine](https://discuss.huggingface.co/u/Ludivine)\
**Post date:** [May 12, 2023, 7:09am UTC](https://discuss.huggingface.co/t/streaming-output-text-when-deploying-on-sagemaker/39611/1 "2023-05-12T07:09:49Z")

</div>

Hi,  
I’m working on fine-tuning bloom and deploying the model on Sagemaker and I wanted to know if it’s possible to stream the output generated text, with directly modifying the inference functions?  
I already tried with applying the TextIteratorStreamer within predict\_fn function, but this doesn’t seem to be the solution.  
Any idea about this would be really appreciate 😊

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [May 12, 2023, 1:58pm UTC](https://discuss.huggingface.co/t/streaming-output-text-when-deploying-on-sagemaker/39611/2 "2023-05-12T13:58:44Z")

</div>

SageMaker is currently not supporting streaming responses.

---

<div class="post-metadata">

**Author:** ![Ludivine](https://avatars.discourse-cdn.com/v4/letter/l/ecae2f/32.png) [@Ludivine](https://discuss.huggingface.co/u/Ludivine)\
**Post date:** [May 12, 2023, 2:39pm UTC](https://discuss.huggingface.co/t/streaming-output-text-when-deploying-on-sagemaker/39611/3 "2023-05-12T14:39:22Z")

</div>

Ok thank you @philschmid

---

<div class="post-metadata">

**Author:** ![RemiP](https://avatars.discourse-cdn.com/v4/letter/r/ebca7d/32.png) [@RemiP](https://discuss.huggingface.co/u/RemiP)\
**Post date:** [May 16, 2023, 3:06pm UTC](https://discuss.huggingface.co/t/streaming-output-text-when-deploying-on-sagemaker/39611/4 "2023-05-16T15:06:34Z")

</div>

Hi @Ludivine , hi @philschmid ,

I’m also trying to stream in a GPT like style the output of my LLM. I’m trying to use https request and an iterator streamer. However, it would be great if it was a built-in possibility with huggingface/sagemaker.

Thanks for your great job.

---

<div class="post-metadata">

**Author:** ![Harsh1729](https://avatars.discourse-cdn.com/v4/letter/h/51bf81/32.png) [@Harsh1729](https://discuss.huggingface.co/u/Harsh1729)\
**Post date:** [June 11, 2023, 9:55am UTC](https://discuss.huggingface.co/t/streaming-output-text-when-deploying-on-sagemaker/39611/5 "2023-06-11T09:55:22Z")

</div>

Hey @RemiP thanks for your response. Can you pls elaborate how are you streaming outputs from the LLM deployed as HuggingFace inference endpoint? Appreciate you help:)

---

<div class="post-metadata">

**Author:** ![aastha6](https://avatars.discourse-cdn.com/v4/letter/a/7bcc69/32.png) [@aastha6](https://discuss.huggingface.co/u/aastha6)\
**Post date:** [October 6, 2023, 9:48am UTC](https://discuss.huggingface.co/t/streaming-output-text-when-deploying-on-sagemaker/39611/6 "2023-10-06T09:48:17Z")

</div>

Sagemaker real-time inference endpoints now supports streaming. Check this [blog](https://aws.amazon.com/blogs/machine-learning/elevating-the-generative-ai-experience-introducing-streaming-support-in-amazon-sagemaker-hosting/)