# Guide/Tutorial to write an inference endpoint for custom models **URL:** \ **Category:** Inference Endpoints on the Hub\ **Created:** [March 21, 2024, 6:00pm UTC](https://discuss.huggingface.co/t/guide-tutorial-to-write-an-inference-endpoint-for-custom-models/78517 "2024-03-21T18:00:06Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![alvations](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alvations/32/97_2.png) [@alvations](https://discuss.huggingface.co/u/alvations)\ **Post date:** [March 21, 2024, 6:00pm UTC](https://discuss.huggingface.co/t/guide-tutorial-to-write-an-inference-endpoint-for-custom-models/78517/1 "2024-03-21T18:00:06Z")
## Are there any docs/guides/tutorials to write custom endpoints to be hosted on Huggingface Hub’s inference endpoints? We’re particularly looking at these models which does things a little differently from normal Huggingface AutoModels (that’ll need A100): - [Unbabel/XCOMET-XXL · Hugging Face](https://huggingface.co/Unbabel/XCOMET-XXL) - [google/metricx-23-xxl-v2p0 · Hugging Face](https://huggingface.co/google/metricx-23-xxl-v2p0) - [GitHub - google-research/metricx](https://github.com/google-research/metricx) - [GitHub - Unbabel/COMET: A Neural Framework for MT Evaluation](https://github.com/Unbabel/COMET) And potentially also these that runs on T4: - [GitHub - unslothai/unsloth: Fine-tuning & Reinforcement Learning for LLMs. 🦥 Train OpenAI gpt-oss, DeepSeek-R1, Qwen3, Gemma 3, TTS 2x faster with 70% less VRAM.](https://github.com/unslothai/unsloth) ---
**Author:** ![Saran12](https://avatars.discourse-cdn.com/v4/letter/s/b9bd4f/32.png) [@Saran12](https://discuss.huggingface.co/u/Saran12)\ **Post date:** [May 16, 2024, 5:27am UTC](https://discuss.huggingface.co/t/guide-tutorial-to-write-an-inference-endpoint-for-custom-models/78517/2 "2024-05-16T05:27:09Z")
Hello, have you found any guides that were useful to you? We need to write a handler.py file for the mistral7b model that we finetuned using unsloth to deploy on an inference endpoint. ---
**Author:** ![DisgustingOzil](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/disgustingozil/32/28076_2.png) [@DisgustingOzil](https://discuss.huggingface.co/u/DisgustingOzil)\ **Post date:** [June 11, 2024, 3:49am UTC](https://discuss.huggingface.co/t/guide-tutorial-to-write-an-inference-endpoint-for-custom-models/78517/3 "2024-06-11T03:49:48Z")
Hi @Saran12 Have you found any solution for writing handler.py for unsloth finetuned mistral 7b? I am facing same issue. ---
**Author:** ![Saran12](https://avatars.discourse-cdn.com/v4/letter/s/b9bd4f/32.png) [@Saran12](https://discuss.huggingface.co/u/Saran12)\ **Post date:** [June 11, 2024, 5:51am UTC](https://discuss.huggingface.co/t/guide-tutorial-to-write-an-inference-endpoint-for-custom-models/78517/4 "2024-06-11T05:51:20Z")
Hi @DisgustingOzil, we had the error due to incorrect procedure, instead of merging the adapter weights with the base model, we just uploaded the adapter weights directly and tried to deploy that instead. After reading the following , [Config.json is not saving after finetuning Llama 2 - #5 by hemanthkumar23](https://discuss.huggingface.co/t/config-json-is-not-saving-after-finetuning-llama-2/56871/5), by using the merge\_and\_unload() function, we were able to upload the complete model that was deployable. Unfortunately I don’t know how to reduce the size of the merged model. Hope this helps! ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [June 11, 2024, 7:00am UTC](https://discuss.huggingface.co/t/guide-tutorial-to-write-an-inference-endpoint-for-custom-models/78517/5 "2024-06-11T07:00:28Z")
Hi, This blog post shows how to deploy any custom model: [Custom Inference with Hugging Face Inference Endpoints](https://www.philschmid.de/custom-inference-handler). ---
**Author:** ![codewithRiz](https://avatars.discourse-cdn.com/v4/letter/c/85f322/32.png) [@codewithRiz](https://discuss.huggingface.co/u/codewithRiz)\ **Post date:** [October 19, 2024, 6:54am UTC](https://discuss.huggingface.co/t/guide-tutorial-to-write-an-inference-endpoint-for-custom-models/78517/6 "2024-10-19T06:54:13Z")
this help me to create some custom handler.py file for my project whis is not in pure transformer > **[Create custom Inference Handler](https://huggingface.co/docs/inference-endpoints/en/guides/custom_handler#3-customize-endpointhandler)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science.