# Error while trying to host finetuned model on inference endpoint

**URL:** <https://discuss.huggingface.co/t/error-while-trying-to-host-finetuned-model-on-inference-endpoint/87462>\
**Category:** Inference Endpoints on the Hub\
**Created:** [May 20, 2024, 5:53am UTC](https://discuss.huggingface.co/t/error-while-trying-to-host-finetuned-model-on-inference-endpoint/87462 "2024-05-20T05:53:35Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Saran12](https://avatars.discourse-cdn.com/v4/letter/s/b9bd4f/32.png) [@Saran12](https://discuss.huggingface.co/u/Saran12)\
**Post date:** [May 20, 2024, 5:53am UTC](https://discuss.huggingface.co/t/error-while-trying-to-host-finetuned-model-on-inference-endpoint/87462/1 "2024-05-20T05:53:35Z")

</div>

Hi,

I have model that was finetuned using unsloth, I’m trying to host the model on an inference endpoint, but i end up getting the following error when trying to initialise the model on a t4 GPU with 64 gb ram

`8:50.284772Z","level":"INFO","fields":{"message":"Waiting for shard to gracefully shutdown"},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.286334Z","level":"INFO","fields":{"message":"Terminating shard"},"target":"text_generation_launcher","span":{"rank":2,"name":"shard-manager"},"spans":[{"rank":2,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.286333Z","level":"INFO","fields":{"message":"Terminating shard"},"target":"text_generation_launcher","span":{"rank":3,"name":"shard-manager"},"spans":[{"rank":3,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.286883Z","level":"INFO","fields":{"message":"Waiting for shard to gracefully shutdown"},"target":"text_generation_launcher","span":{"rank":2,"name":"shard-manager"},"spans":[{"rank":2,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.286915Z","level":"INFO","fields":{"message":"Waiting for shard to gracefully shutdown"},"target":"text_generation_launcher","span":{"rank":3,"name":"shard-manager"},"spans":[{"rank":3,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.688669Z","level":"INFO","fields":{"message":"shard terminated"},"target":"text_generation_launcher","span":{"rank":3,"name":"shard-manager"},"spans":[{"rank":3,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.879857Z","level":"INFO","fields":{"message":"shard terminated"},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:50.886106Z","level":"INFO","fields":{"message":"shard terminated"},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]} {"timestamp":"2024-05-20T05:28:51.088917Z","level":"INFO","fields":{"message":"shard terminated"},"target":"text_generation_launcher","span":{"rank":2,"name":"shard-manager"},"spans":[{"rank":2,"name":"shard-manager"}]} Error: WebserverFailed`

here is a link to the repository

> **[Saran12/Blusapphire\_AI · Hugging Face](https://huggingface.co/Saran12/Blusapphire_AI)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Is it due to erroneous settings in the inference endpoint?, or was the repository uploaded missing something? Any help would be appreciated.

---

<div class="post-metadata">

**Author:** ![meganariley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/meganariley/32/56966_2.png) [@meganariley](https://discuss.huggingface.co/u/meganariley)\
**Post date:** [May 21, 2024, 3:59pm UTC](https://discuss.huggingface.co/t/error-while-trying-to-host-finetuned-model-on-inference-endpoint/87462/2 "2024-05-21T15:59:04Z")

</div>

Hi @Saran12 Can you please try using this model with an A10G or A100 instance instead of the T4? Let us know if you continue experiencing issues!

---

<div class="post-metadata">

**Author:** ![Saran12](https://avatars.discourse-cdn.com/v4/letter/s/b9bd4f/32.png) [@Saran12](https://discuss.huggingface.co/u/Saran12)\
**Post date:** [May 22, 2024, 5:42am UTC](https://discuss.huggingface.co/t/error-while-trying-to-host-finetuned-model-on-inference-endpoint/87462/3 "2024-05-22T05:42:11Z")

</div>

Yup it is working on A10G, thank you!