# Accelerated Inference API not taking parameters?

**URL:** <https://discuss.huggingface.co/t/accelerated-inference-api-not-taking-parameters/17377>\
**Category:** Intermediate\
**Created:** [May 1, 2022, 3:04am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-not-taking-parameters/17377 "2022-05-01T03:04:43Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![juancavallotti](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/juancavallotti/32/4976_2.png) [@juancavallotti](https://discuss.huggingface.co/u/juancavallotti)\
**Post date:** [May 1, 2022, 3:04am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-not-taking-parameters/17377/1 "2022-05-01T03:04:43Z")

</div>

Hello! I’m trying to generate text using a fine-tuned T5, and I’m running into some truncation issues.

From the docs, I can see that if I send the parameter max\_new\_tokens I can potentially get longer answers, but the API is responding always in the same length no matter what I do. Here is the payload I’m sending. I also tried wiggling the parameter names and so, the API is validating unknown parameters responding 400, but it keeps truncating the response when I send what it looks like a correct request.

If I use the model in transformers, I get longer responses.

Here is what I’m sending.

```javascript
const inference_endpoint = "https://api-inference.huggingface.co/models/squidgy/t5-ml-finetuned-gec"

    headers: {
      "Authorization": "Bearer " + process.env.HF_TOKEN,
      "Content-Type": "application/json"
    },
    url: inference_endpoint,
    method: "post",
    data: {
      inputs: query,
      parameters: {
        max_new_tokens: 196,
      },
      options: {
        wait_for_model: await_for_model
      }
    }

```

Am I doing anything wrong? Thanks in advance for your help!!

---

<div class="post-metadata">

**Author:** ![Noplease](https://avatars.discourse-cdn.com/v4/letter/n/a87d85/32.png) [@Noplease](https://discuss.huggingface.co/u/Noplease)\
**Post date:** [September 22, 2022, 5:13am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-not-taking-parameters/17377/2 "2022-09-22T05:13:42Z")

</div>

Hi  
Same problem  
How can one define max lenght output size in an inference API ?  
Thanks

---

<div class="post-metadata">

**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.huggingface.co/u/kernelpanic)\
**Post date:** [September 22, 2022, 8:16am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-not-taking-parameters/17377/3 "2022-09-22T08:16:14Z")

</div>

Hi @juancavallotti,  
Did you try ‘max\_length’ instead of ‘max\_new\_tokens’?

---

<div class="post-metadata">

**Author:** ![juancavallotti](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/juancavallotti/32/4976_2.png) [@juancavallotti](https://discuss.huggingface.co/u/juancavallotti)\
**Post date:** [September 22, 2022, 2:30pm UTC](https://discuss.huggingface.co/t/accelerated-inference-api-not-taking-parameters/17377/4 "2022-09-22T14:30:54Z")

</div>

Yup the parameter didn’t get rejected but didn’t work either

---

<div class="post-metadata">

**Author:** ![SarcasticApe](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sarcasticape/32/12091_2.png) [@SarcasticApe](https://discuss.huggingface.co/u/SarcasticApe)\
**Post date:** [October 18, 2022, 9:21am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-not-taking-parameters/17377/5 "2022-10-18T09:21:09Z")

</div>

any update/solution on this issue ? i’m encountering the same problem with the bloom model, none of the advanced parameters of the inference API seems to work.

---

<div class="post-metadata">

**Author:** ![uccollab](https://avatars.discourse-cdn.com/v4/letter/u/58956e/32.png) [@uccollab](https://discuss.huggingface.co/u/uccollab)\
**Post date:** [October 26, 2022, 10:25am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-not-taking-parameters/17377/6 "2022-10-26T10:25:19Z")

</div>

Same problem for me and I didn’t receive any reply in my thread. It seems like the Inference API for BLOOM is basically broken and only allows basic generation.

Personally, I’ve found the following parameters being ignored:

- max\_new\_tokens
- temperature
- do\_sample
- use\_gpu (but this is to be expected, afaik HF handles GPU on Inference API with a separate pricing)

do\_sample is particularly frustrating as BLOOM generates the same output over and over for a given input. I managed to force a bit of variation by playing with top\_k, but this is not very rigorous.