# SageMaker Inference for Model Tuned Elsewhere

**URL:** <https://discuss.huggingface.co/t/sagemaker-inference-for-model-tuned-elsewhere/9606>\
**Category:** Amazon SageMaker\
**Created:** [August 30, 2021, 5:28pm UTC](https://discuss.huggingface.co/t/sagemaker-inference-for-model-tuned-elsewhere/9606 "2021-08-30T17:28:40Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![charlesatftl](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@charlesatftl](https://discuss.huggingface.co/u/charlesatftl)\
**Post date:** [August 30, 2021, 5:28pm UTC](https://discuss.huggingface.co/t/sagemaker-inference-for-model-tuned-elsewhere/9606/1 "2021-08-30T17:28:40Z")

</div>

Reading through the documentation for HuggingFace & SageMaker as we are evaluating it and found the following:

> Q: Which models can I deploy for Inference?
> 
> A: You can deploy
> 
> - any 🤗 Transformers model trained in Amazon SageMaker, or other compatible platforms and that can accomodate the SageMaker Hosting design
> - any of the 10 000+ publicly available Transformer models from the Hugging Face [Model Hub](https://huggingface.co/models), or
> - your private models hosted in your Hugging Face premium account!

Is it possible to fine-tune a model elsewhere, outside of SageMaker Training, (for instance, just through a regular PyTorch training loop on a pretrained `transformers` model), and then deploy it for Inference without hosting it on an account?

Would appreciate any pointers y’all can give on this.

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [August 30, 2021, 5:42pm UTC](https://discuss.huggingface.co/t/sagemaker-inference-for-model-tuned-elsewhere/9606/2 "2021-08-30T17:42:37Z")

</div>

Hello @charlesatftl,

Yes, you can fine-tune `transformers` anywhere you want and use it in SageMaker for Inference.

There are currently two options to use your model then:

1. Push your model to [Models - Hugging Face](http://hf.co/models) and deploy it directory from there, see: [Deploy models to Amazon SageMaker](https://huggingface.co/docs/sagemaker/inference#deploy-for-inference-one-of-the-10000-%F0%9F%A4%97-transformers-models-available-in-the-%F0%9F%A4%97-hub)
2. Create a `model.tar.gz` upload it to s3 and deploy it from there, see: [Deploy models to Amazon SageMaker](https://huggingface.co/docs/sagemaker/inference#deploy-the-model-using-model_data)  
In Addition to 2. here is the documentation on how to create a `model.tar.gz` [Deploy models to Amazon SageMaker](https://huggingface.co/docs/sagemaker/inference#creating-a-model-artifact-modeltargz-for-deployment)

---

<div class="post-metadata">

**Author:** ![charlesatftl](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@charlesatftl](https://discuss.huggingface.co/u/charlesatftl)\
**Post date:** [August 30, 2021, 6:16pm UTC](https://discuss.huggingface.co/t/sagemaker-inference-for-model-tuned-elsewhere/9606/3 "2021-08-30T18:16:12Z")

</div>

Thank you! This is very helpful.

---

<div class="post-metadata">

**Author:** ![charlesatftl](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@charlesatftl](https://discuss.huggingface.co/u/charlesatftl)\
**Post date:** [September 2, 2021, 3:06pm UTC](https://discuss.huggingface.co/t/sagemaker-inference-for-model-tuned-elsewhere/9606/4 "2021-09-02T15:06:06Z")

</div>

Now running into issues when I attempt to use a model as you suggested with a modified `model.tar.gz`. Specifically, when I add a `code` directory with `inference.py` and `requirements.txt`, it starts throwing errors even if those are both empty files, saying that it cannot find the config. The same model with the `code` directory removed works properly. Error is along the lines of “Can’t load config for … Make sure that: … is a correct model identifier listed on [Models - Hugging Face](https://huggingface.co/models)… or … is the correct path to a directory containing a config.json file.”

---

<div class="post-metadata">

**Author:** ![charlesatftl](https://avatars.discourse-cdn.com/v4/letter/c/6f9a4e/32.png) [@charlesatftl](https://discuss.huggingface.co/u/charlesatftl)\
**Post date:** [September 2, 2021, 3:32pm UTC](https://discuss.huggingface.co/t/sagemaker-inference-for-model-tuned-elsewhere/9606/5 "2021-09-02T15:32:19Z")

</div>

Update: I believe this was an error on my end. You’ve gotta be careful to `tar` the model with the right flags or else the archive won’t have the right structure (i.e. it’ll have the files nested in some relative path). 🙂