# Deploying custom inference script with llama2 finetuned model

**URL:** <https://discuss.huggingface.co/t/deploying-custom-inference-script-with-llama2-finetuned-model/63280>\
**Category:** Amazon SageMaker\
**Created:** [November 23, 2023, 12:00pm UTC](https://discuss.huggingface.co/t/deploying-custom-inference-script-with-llama2-finetuned-model/63280 "2023-11-23T12:00:33Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![saitejinfrrd](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@saitejinfrrd](https://discuss.huggingface.co/u/saitejinfrrd)\
**Post date:** [November 23, 2023, 12:00pm UTC](https://discuss.huggingface.co/t/deploying-custom-inference-script-with-llama2-finetuned-model/63280/1 "2023-11-23T12:00:33Z")

</div>

Hello

I have used the following code to deploy fine tuned llama2 model with a custom inference script. the problem I have is that the custom `inference.py` script is not being used. I have made changes to the `predict_fn` and `model_fn` but it returns the usual response.  
Here’s my code:

```auto
config = {
  'HF_MODEL_ID': "/opt/ml/model", 
  'SM_NUM_GPUS': json.dumps(1), # Number of GPU used per replica
  'MAX_INPUT_LENGTH': json.dumps(2048), # Max length of input text
  'MAX_TOTAL_TOKENS': json.dumps(4096), # Max length of the generation (including input text)

}

```

```auto
hf_model = HuggingFaceModel(
        model_data=s3_model_uri,
        role=role,
    image_uri=llm_image,
    env=config

    )

```

The s3\_model\_uri holds the model artifacts along with the inference.py code.The files are as follows:

```auto
model.tar.gz
|-- model artifacts
|-- code
| |-- inference.py
| |-- generate.py

```

The llm\_image i get from `get_huggingface_llm_image_uri` is:  
`llm image uri: 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-tgi-inference:2.0.1-tgi1.0.3-gpu-py39-cu118-ubuntu20.04`

Any help on this would be appreciated.  
Thanks

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [November 23, 2023, 1:22pm UTC](https://discuss.huggingface.co/t/deploying-custom-inference-script-with-llama2-finetuned-model/63280/2 "2023-11-23T13:22:29Z")

</div>

You cannot use the llm container with a custom inference script.

---

<div class="post-metadata">

**Author:** ![saitejinfrrd](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@saitejinfrrd](https://discuss.huggingface.co/u/saitejinfrrd)\
**Post date:** [November 23, 2023, 1:40pm UTC](https://discuss.huggingface.co/t/deploying-custom-inference-script-with-llama2-finetuned-model/63280/3 "2023-11-23T13:40:32Z")

</div>

I have modified and provided the following as suggested with out any modifications to the above config :

```auto
hf_model = HuggingFaceModel(
    model_data=s3_model_uri,
    role=role,

    env=config,
    transformers_version="4.28.1",
    pytorch_version="2.0.0",
    py_version="py310",
    )

```

I get the following error in the logs:

```auto

huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/opt/ml/model'. Use `repo_type` argument if needed.

```

I understand that I should change `HF_MODEL_ID` but if give the repo\_name from the huggingface hub, will it consider the artifacts in the tar.gz file ?

---

<div class="post-metadata">

**Author:** ![saitejinfrrd](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@saitejinfrrd](https://discuss.huggingface.co/u/saitejinfrrd)\
**Post date:** [November 24, 2023, 11:08am UTC](https://discuss.huggingface.co/t/deploying-custom-inference-script-with-llama2-finetuned-model/63280/4 "2023-11-24T11:08:27Z")

</div>

Even after providing a hf\_hub id, it does take the model from the tar.gz file but it searches for a .bin file. I am currently having the model in safetensors format so is there a hack to get pass this?

`2023-11-24T11:00:07,577 [INFO] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Failed to load the model: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /opt/ml/model.`

---

<div class="post-metadata">

**Author:** ![Dnsibu](https://avatars.discourse-cdn.com/v4/letter/d/f1d935/32.png) [@Dnsibu](https://discuss.huggingface.co/u/Dnsibu)\
**Post date:** [December 14, 2023, 9:51pm UTC](https://discuss.huggingface.co/t/deploying-custom-inference-script-with-llama2-finetuned-model/63280/5 "2023-12-14T21:51:39Z")

</div>

what i did is just convert the safetensors file into a .bin file

import safetensors  
import torch  
model\_path=“path/to/your/model/.safetensors”  
pt\_state\_dict = safetensors.torch.load\_file(model\_path, device=“cpu”)  
torch.save(pt\_state\_dict, “pytorch\_model.bin”)

---

<div class="post-metadata">

**Author:** ![sourceful-tianlin](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@sourceful-tianlin](https://discuss.huggingface.co/u/sourceful-tianlin)\
**Post date:** [January 2, 2024, 5:49pm UTC](https://discuss.huggingface.co/t/deploying-custom-inference-script-with-llama2-finetuned-model/63280/6 "2024-01-02T17:49:23Z")

</div>

Is it the only fix to this problem? ?

---

<div class="post-metadata">

**Author:** ![Dnsibu](https://avatars.discourse-cdn.com/v4/letter/d/f1d935/32.png) [@Dnsibu](https://discuss.huggingface.co/u/Dnsibu)\
**Post date:** [January 4, 2024, 5:44pm UTC](https://discuss.huggingface.co/t/deploying-custom-inference-script-with-llama2-finetuned-model/63280/7 "2024-01-04T17:44:48Z")

</div>

So far that’s what I’ve been only able to find.