# Accelerated Inference API can't load a model on GPU **URL:** \ **Category:** Intermediate\ **Created:** [August 29, 2022, 5:37am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248 "2022-08-29T05:37:59Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.huggingface.co/u/kernelpanic)\ **Post date:** [August 29, 2022, 5:37am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/1 "2022-08-29T05:37:59Z")
Hi, I am trying to run models via accelerated inference API using GPU. I already subscribed Community Pro plan in order to use GPU. The inference API works fine if I use CPU. However, the API cannot load models on GPU. I used the following code (where MY\_API\_TOKEN is replaced by my actual API token string): > import requests > import json > API\_TOKEN = ‘MY\_API\_TOKEN’ > API\_URL = “[https://api-inference.huggingface.co/models/Salesforce/codegen-350M-multi](https://api-inference.huggingface.co/models/Salesforce/codegen-350M-multi)” > headers = {“Authorization”: f"Bearer {API\_TOKEN}"} > payload = {‘inputs’: ‘def hello\_world():’, ‘options’: {‘use\_gpu’: True}} > data = json.dumps(payload) > response = requests.request(“POST”, API\_URL, headers=headers, data=data) > post\_result = json.loads(response.content.decode(“utf-8”)) > print(post\_result) Then I get the message below no matter how long I wait: > {‘error’: ‘Model Salesforce/codegen-350M-multi is currently loading’, ‘estimated\_time’: 31.894704818725586} What’s wrong with the accelerated inference API? How can I load models on GPU? ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [August 29, 2022, 4:23pm UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/2 "2022-08-29T16:23:17Z")
cc @Narsil ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.huggingface.co/u/kernelpanic)\ **Post date:** [September 7, 2022, 8:41am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/3 "2022-09-07T08:41:02Z")
@nielsr Thanks for your attention to this topic. I still can’t load a model on GPU. Meanwhile, I tried to use pinned models using CPU (since it looks like using CPU is fast enough once the model is loaded), but I think my pinned models are not preloaded becase I get the error message (‘error’: ‘…model is currently loading’) for the first inference. Could you look into these issues? (GPU and pinned models) ---
**Author:** ![michellehbn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/michellehbn/32/58964_2.png) [@michellehbn](https://discuss.huggingface.co/u/michellehbn)\ **Post date:** [September 7, 2022, 3:27pm UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/4 "2022-09-07T15:27:39Z")
Hi @kernelpanic ! Thanks for reaching out and letting us know about the issue and error message. We’ll take a look to see what’s going on and follow-up soon. Thanks again! Michelle ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.huggingface.co/u/kernelpanic)\ **Post date:** [September 22, 2022, 7:21am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/5 "2022-09-22T07:21:49Z")
Hi @michellehbn . Today I received a bill including $120 for Pinned Models on CPU and $5 for Pinned Models on GPU, which I could never use as mentioned above ☹ Can I get a refund on these pinned models? ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.huggingface.co/u/kernelpanic)\ **Post date:** [September 22, 2022, 7:29am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/6 "2022-09-22T07:29:21Z")
Also, I get charged $0.02 for Accelerated Inference API on GPU. I don’t understand why this is included in the bill because (1) The description of accelerated inference API looks like there is no extra charge for GPU if I subscribe Community Pro plan, and - see “Accelerated inference for a number of supported models on CPU and GPU (GPU requires a Community Pro or Organization Lab plan)” in [Overview](https://huggingface.co/docs/api-inference/quicktour) (2) I could not use the inference API on GPU as I mentioned in the first post. Could you explain why $0.02 for Accelerated Inference API on GPU is included in the bill? ---
**Author:** ![michellehbn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/michellehbn/32/58964_2.png) [@michellehbn](https://discuss.huggingface.co/u/michellehbn)\ **Post date:** [September 22, 2022, 9:07am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/7 "2022-09-22T09:07:27Z")
Hi @kernelpanic, Pricing for the Inference API is based on the number of characters going through the endpoint: you’ll have up to 30k input characters /month for free (on CPU). Pro plan users will have up to 1M input characters /month for text tasks and up to 2h of audio for audio tasks, then pay-as-you-go with $10/M characters on CPU, $50/M characters on GPU. When you pin a model on CPU it’s $1/day or on GPU it will be $5/day with one day started being one day due. Would you still like to us to continue working on loading ‘Salesforce/codegen-350M-multi’ on GPU? Sorry to hear about the pinned models you were unable to use, would you mind sending an email to [api-enterprise@huggingface.co](mailto:api-enterprise@huggingface.co) with the invoice or invoice number please? Thanks so much! ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.huggingface.co/u/kernelpanic)\ **Post date:** [September 22, 2022, 10:30am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/8 "2022-09-22T10:30:52Z")
Thank you so much for the kind reply @michellehbn. The current pricing page has no information about what you explained. I think it should be clearly described in the pricing page. (Please let me know if there is an official pricing page about this.) I would still like you to continue working on loading the model on GPU. The reason why I asked for a refund is because I couldn’t use it (not because I don’t need it…). I sent an email to [api-enterprise@huggingface.co](mailto:api-enterprise@huggingface.co). I will post an update as soon as I receive a response. Thanks again. ---
**Author:** ![michellehbn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/michellehbn/32/58964_2.png) [@michellehbn](https://discuss.huggingface.co/u/michellehbn)\ **Post date:** [September 22, 2022, 12:35pm UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/9 "2022-09-22T12:35:26Z")
Sure thing @kernelpanic ! I haven’t seen the email come through just yet, do you mind sending to me at [michelle@huggingface.co](mailto:michelle@huggingface.co) as well please? Sorry for that, I’ll help follow this through. ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.huggingface.co/u/kernelpanic)\ **Post date:** [September 23, 2022, 1:20am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/10 "2022-09-23T01:20:38Z")
Hi @michellehbn, I sent another email to [michelle@huggingface.co](mailto:michelle@huggingface.co) with the invoice number. Thanks for your support and help. ---
**Author:** ![michellehbn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/michellehbn/32/58964_2.png) [@michellehbn](https://discuss.huggingface.co/u/michellehbn)\ **Post date:** [October 25, 2022, 1:25pm UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/11 "2022-10-25T13:25:20Z")
Hi @kernelpanic! Thanks for bearing with us and sorry for the delay in follow-up! 🤗 We’ve made some changes to the Inference API - this is now free for use, as a solution to easily explore and evaluate models, and Inference Endpoints is our new paid inference solution for production use cases! The free Inference API is subject to rate limiting for heavy use cases. For larger volumes of requests, or if you need guaranteed latency/performance, you can use our new solution [Inference Endpoints](https://huggingface.co/inference-endpoints) to easily [deploy your models](https://ui.endpoints.huggingface.co/new) on dedicated, fully-managed infrastructure. Inference Endpoints will give you the flexibility to quickly create endpoints on CPU or GPU resources, and is billed by compute uptime vs character usage. Further pricing information can be found [here](https://huggingface.co/docs/inference-endpoints/pricing). We’re all ears for feedback so please let us know what you think! Thanks so much! ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.huggingface.co/u/kernelpanic)\ **Post date:** [October 26, 2022, 1:40am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/12 "2022-10-26T01:40:38Z")
Hi @michellehbn. Thank you very much for letting me know the update. It is amazing that you made the Inference API free for use. I will take a look at the new inference solution as well. Thank you and have a nice day! ---
**Author:** ![metkor](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/metkor/32/13372_2.png) [@metkor](https://discuss.huggingface.co/u/metkor)\ **Post date:** [January 5, 2023, 12:46pm UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/13 "2023-01-05T12:46:07Z")
Hello @michellehbn, Is this still valid for Inference API: With Pro account for 1M character per month and if we hit this limit in the **same month** we will pay $10 for each M for our usage? Or do we have to migrate Inference Endpoints? ---
**Author:** ![michellehbn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/michellehbn/32/58964_2.png) [@michellehbn](https://discuss.huggingface.co/u/michellehbn)\ **Post date:** [January 16, 2023, 11:06am UTC](https://discuss.huggingface.co/t/accelerated-inference-api-cant-load-a-model-on-gpu/22248/14 "2023-01-16T11:06:55Z")
Hi @metkor! Happy new year! Sorry for my delay in response. The PRO subscription will give you higher Inference API rate limits than the free Inference API plan. The pricing mentioned earlier ($10/M characters, etc) is no longer valid. If you need guaranteed latency/performance, you should use our new solution [Inference Endpoints](https://huggingface.co/inference-endpoints) to easily [deploy your models](https://ui.endpoints.huggingface.co/new) on dedicated, fully-managed infrastructure, and is billed by compute uptime vs character usage. Further pricing information can be found [here](https://huggingface.co/docs/inference-endpoints/pricing). Please let us know if there are any other questions! Thanks again!