# How does the GPT-J inference API work?

**URL:** <https://discuss.huggingface.co/t/how-does-the-gpt-j-inference-api-work/10337>\
**Category:** Beginners\
**Created:** [September 27, 2021, 2:58pm UTC](https://discuss.huggingface.co/t/how-does-the-gpt-j-inference-api-work/10337 "2021-09-27T14:58:21Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![wittyseller](https://avatars.discourse-cdn.com/v4/letter/w/c4cdca/32.png) [@wittyseller](https://discuss.huggingface.co/u/wittyseller)\
**Post date:** [September 27, 2021, 2:58pm UTC](https://discuss.huggingface.co/t/how-does-the-gpt-j-inference-api-work/10337/1 "2021-09-27T14:58:21Z")

</div>

Hi All.  
I started a 7 day trial for the startup plan. I need to use GPT-J through HF inference API. I pinned it in on org account to work on GPU and, after sending a request, all I get back is a single generated word. The max token param is set to 100.  
Could you please let me know how should I make it generate more than one word 🙂

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [September 28, 2021, 7:46am UTC](https://discuss.huggingface.co/t/how-does-the-gpt-j-inference-api-work/10337/2 "2021-09-28T07:46:19Z")

</div>

Hi,

Normally it should become available. I’ll ask the team and get back to you.

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [September 28, 2021, 8:03am UTC](https://discuss.huggingface.co/t/how-does-the-gpt-j-inference-api-work/10337/3 "2021-09-28T08:03:16Z")

</div>

GPT-J is supported by the inference API. You can try it out here: [EleutherAI/gpt-j-6B · Hugging Face](https://huggingface.co/EleutherAI/gpt-j-6B?text=This+is+a+test+to)

Can you verify whether you are able to generate more than a given word?

---

<div class="post-metadata">

**Author:** ![wittyseller](https://avatars.discourse-cdn.com/v4/letter/w/c4cdca/32.png) [@wittyseller](https://discuss.huggingface.co/u/wittyseller)\
**Post date:** [September 29, 2021, 7:03am UTC](https://discuss.huggingface.co/t/how-does-the-gpt-j-inference-api-work/10337/4 "2021-09-29T07:03:39Z")

</div>

Thanks for your reply! I verified and it still generates only one word for me. I’m sending the below mentioned json in my request to [https://api-inference.huggingface.co/models/EleutherAI/gpt-j-6B](https://api-inference.huggingface.co/models/EleutherAI/gpt-j-6B)

{  
“inputs”: “Nowadays, many users would like to upgrade old hard drive to SSD with Windows installed, or reinstall Windows 10 on SSD afterward. The faster boot speed and reading & writing speed make it known as a better boot drive.\r\n\r\nTo be specifc, after installing the operating system on a new SSD, you’ll find the computer boots up faster, and runs smoothly even with muliple programs in the background.\r\n\r\nSo let’s”,  
“parameters”: {  
“return\_full\_text”: false,  
“max\_new\_tokens”: 100,  
“temperature”: 0.8  
},  
“options”: {  
“use\_cache”: false  
}  
}

And I’m getting back this:

HTTP/1.1 200 OK  
date: Wed, 29 Sep 2021 06:56:12 GMT,Wed, 29 Sep 2021 06:56:21 GMT  
server: istio-envoy  
x-compute-time: 0.1596  
x-compute-type: gpu  
access-control-expose-headers: x-compute-type, x-compute-time  
x-compute-characters: 406  
content-length: 27  
content-type: application/json  
x-envoy-upstream-service-time: 177

[{“generated\_text”:" get"}]

Is something wrong on my side?

---

<div class="post-metadata">

**Author:** ![wittyseller](https://avatars.discourse-cdn.com/v4/letter/w/c4cdca/32.png) [@wittyseller](https://discuss.huggingface.co/u/wittyseller)\
**Post date:** [September 29, 2021, 7:21am UTC](https://discuss.huggingface.co/t/how-does-the-gpt-j-inference-api-work/10337/5 "2021-09-29T07:21:47Z")

</div>

> [@wittyseller](#):
>
> Nowadays, many users would like to upgrade old hard drive to SSD with Windows installed, or reinstall Windows 10 on SSD afterward.

It’s strange: if I send only one sentence as input, it works:

So this is my request:  
{  
“inputs”: “Nowadays, many users would like to upgrade old hard drive to SSD with Windows installed, or reinstall Windows 10 on SSD afterward.”,  
“parameters”: {  
“return\_full\_text”: false,  
“max\_new\_tokens”: 100,  
“temperature”: 0.8  
},  
“options”: {  
“use\_cache”: false  
}  
}

And I get back this:

HTTP/1.1 200 OK  
date: Wed, 29 Sep 2021 07:06:20 GMT,Wed, 29 Sep 2021 07:06:29 GMT  
server: istio-envoy  
x-compute-time: 2.1995999999999998  
x-compute-type: gpu  
access-control-expose-headers: x-compute-type, x-compute-time  
x-compute-characters: 130  
content-length: 130  
content-type: application/json  
x-envoy-upstream-service-time: 2244

[{“generated\_text”:" In this way, they can enjoy the performance of SSD and the convenience of Windows. In the first place, the"}]

To me, it looks like there is some sort of a limit on the input tokens

---

<div class="post-metadata">

**Author:** ![kaisar123](https://avatars.discourse-cdn.com/v4/letter/k/6f9a4e/32.png) [@kaisar123](https://discuss.huggingface.co/u/kaisar123)\
**Post date:** [October 8, 2021, 3:48pm UTC](https://discuss.huggingface.co/t/how-does-the-gpt-j-inference-api-work/10337/6 "2021-10-08T15:48:26Z")

</div>

Try using “max\_length” parameter instead of “max\_new\_tokens”. The documentation suggests they serve the same purpose and you should not use both simultaneously. Worked for me.