# Text Generation response truncation

**URL:** <https://discuss.huggingface.co/t/text-generation-response-truncation/53155>\
**Category:** Beginners\
**Created:** [September 1, 2023, 2:46pm UTC](https://discuss.huggingface.co/t/text-generation-response-truncation/53155 "2023-09-01T14:46:03Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![cCaldwell](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@cCaldwell](https://discuss.huggingface.co/u/cCaldwell)\
**Post date:** [September 1, 2023, 2:46pm UTC](https://discuss.huggingface.co/t/text-generation-response-truncation/53155/1 "2023-09-01T14:46:03Z")

</div>

I am having an issue where all responses are truncated, no matter which model or where they are hosted. Other questions on this topic have referenced required changes in langchain modules, but we are not using langchain.

For reference, I am getting the same error from:  
databricks/dolly-v2-3b run on databricks ML RT 13.3 (56 ram 8cpu GCP)  
databricks/dolly-v2-3b run on HF free inference using huggingface\_hub library  
databricks/dolly-v2-7b huggingface inference endpoint Nvidia A10G using recommended sample input  
tiiuae/falcon-7b huggingface inference endpoint Nvidia A10G using recommended sample input

In case of HF inference endpoints: I get the same error using python requests as I do with the sample ui. Setting for inference endpoint are default: max input tokens: 1024 max tokens:1025. Screenshot below:

 ![truncation](https://us1.discourse-cdn.com/hellohellohello/original/3X/2/9/29acb753cb850fe42cd6406f86ff93d3d3bbef84.png)

---

<div class="post-metadata">

**Author:** ![cCaldwell](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@cCaldwell](https://discuss.huggingface.co/u/cCaldwell)\
**Post date:** [September 1, 2023, 3:04pm UTC](https://discuss.huggingface.co/t/text-generation-response-truncation/53155/2 "2023-09-01T15:04:58Z")

</div>

This problem exists in other modes to. Here is a test on inference endpoint, queried from Databricks(GCP) using requests after setting the inference endpoint to text-to-text:

 ![Screenshot 2023-09-01 at 11.01.24 AM](https://us1.discourse-cdn.com/hellohellohello/original/3X/f/c/fc2ef009c70b6365668091e3c7ce97a46a387619.png)

---

<div class="post-metadata">

**Author:** ![cCaldwell](https://avatars.discourse-cdn.com/v4/letter/c/ad7895/32.png) [@cCaldwell](https://discuss.huggingface.co/u/cCaldwell)\
**Post date:** [September 1, 2023, 3:05pm UTC](https://discuss.huggingface.co/t/text-generation-response-truncation/53155/3 "2023-09-01T15:05:30Z")

</div>

and the same but after setting inference endpoint to summarization mode:

 ![Screenshot 2023-09-01 at 11.03.34 AM](https://us1.discourse-cdn.com/hellohellohello/original/3X/7/2/72dbe995b624b2b4c86863ba679ede6b43a173a0.png)

---

<div class="post-metadata">

**Author:** ![Lucisu](https://avatars.discourse-cdn.com/v4/letter/l/5f8ce5/32.png) [@Lucisu](https://discuss.huggingface.co/u/Lucisu)\
**Post date:** [November 7, 2023, 7:03am UTC](https://discuss.huggingface.co/t/text-generation-response-truncation/53155/4 "2023-11-07T07:03:15Z")

</div>

Hey, @cCaldwell! Have you been able to solve this issue? I have started to have the same problem while using the Llama 2 70b model.

---

<div class="post-metadata">

**Author:** ![santa1666](https://avatars.discourse-cdn.com/v4/letter/s/b2d939/32.png) [@santa1666](https://discuss.huggingface.co/u/santa1666)\
**Post date:** [November 14, 2023, 4:25am UTC](https://discuss.huggingface.co/t/text-generation-response-truncation/53155/5 "2023-11-14T04:25:51Z")

</div>

same issue with 7b

---

<div class="post-metadata">

**Author:** ![iamkmson](https://avatars.discourse-cdn.com/v4/letter/i/aeb1de/32.png) [@iamkmson](https://discuss.huggingface.co/u/iamkmson)\
**Post date:** [June 15, 2024, 5:35pm UTC](https://discuss.huggingface.co/t/text-generation-response-truncation/53155/6 "2024-06-15T17:35:38Z")

</div>

I’m also facing same issue 😑 on all the models I’ve tried.  
Does anyone have the solution for this??

---

<div class="post-metadata">

**Author:** ![PatrikKruse](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrikkruse/32/30631_2.png) [@PatrikKruse](https://discuss.huggingface.co/u/PatrikKruse)\
**Post date:** [August 18, 2024, 7:21pm UTC](https://discuss.huggingface.co/t/text-generation-response-truncation/53155/7 "2024-08-18T19:21:12Z")

</div>

I am using chat completion with Microsofts SemanticKernel and get the same result. Usually the responses are truncated at 100 tokens.

I have two questions, can I raise the bar above 100 tokens and is there a way to programmatically detect that the response are truncated?