# Help for inference.py code

**URL:** <https://discuss.huggingface.co/t/help-for-inference-py-code/15143>\
**Category:** Amazon SageMaker\
**Created:** [February 25, 2022, 5:10pm UTC](https://discuss.huggingface.co/t/help-for-inference-py-code/15143 "2022-02-25T17:10:02Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![GenV](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/genv/32/12132_2.png) [@GenV](https://discuss.huggingface.co/u/GenV)\
**Post date:** [February 25, 2022, 5:10pm UTC](https://discuss.huggingface.co/t/help-for-inference-py-code/15143/1 "2022-02-25T17:10:02Z")

</div>

Hi,  
I’m using the SageMaker / Huggingface inference. For the model.tar.gz requested for the endpoint, I’m using this inference code:

```auto
import os
import torch
from transformers import AutoTokenizer, pipeline, T5Tokenizer

T5_WEIGHTS_NAME = "t5.pt"

def model_fn(model_dir):
    model = torch.load(os.path.join(model_dir, T5_WEIGHTS_NAME))
    tokenizer = T5Tokenizer.from_pretrained(model_dir)

    if torch.cuda.is_available():
        device = 0
    else:
        device = -1

    generation = pipeline("text2text-generation", model=model, tokenizer=tokenizer, device=device, max_length=1024)

    return generation

```

I have worse performance than my local code like this:

```auto
tokenized_text = self.tokenizer_nl(
                input_text, truncation=True, padding="max_length", return_tensors="pt"
            )

source_ids = tokenized_text["input_ids"].to(self.device, dtype=torch.long)
source_mask = tokenized_text["attention_mask"].to(self.device, dtype=torch.long)

generated_ids = self.model.generate(
    input_ids=source_ids, attention_mask=source_mask, max_length=1024
)

pred = self.tokenizer_sql.decode(
    generated_ids[0],
    spaces_between_special_tokens=False,
    skip_special_tokens=True,
)

```

So I want to put this code in my inference.py code, without using the pipeline. But I don’t know how to write this inference. Can someone help me? Thanks!

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [February 25, 2022, 5:34pm UTC](https://discuss.huggingface.co/t/help-for-inference-py-code/15143/2 "2022-02-25T17:34:22Z")

</div>

Hello @GenV,

Which version are you using on SageMaker? and which version are you using locally? For `transformers` and `pytorch`.  
Do you use a GPU on your local machine as well?

---

<div class="post-metadata">

**Author:** ![GenV](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/genv/32/12132_2.png) [@GenV](https://discuss.huggingface.co/u/GenV)\
**Post date:** [February 25, 2022, 7:22pm UTC](https://discuss.huggingface.co/t/help-for-inference-py-code/15143/3 "2022-02-25T19:22:10Z")

</div>

@philschmid I’m using on SageMaker: torch 1.9.1, transformers 4.12.3 and sagemaker 2.77.1

Locally I’m using torch 1.10.2, transformers 4.12.5

Both on GPU.

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [February 28, 2022, 2:56pm UTC](https://discuss.huggingface.co/t/help-for-inference-py-code/15143/4 "2022-02-28T14:56:16Z")

</div>

And what is the latency difference you see? Could test to have the same versions locally as well?

---

<div class="post-metadata">

**Author:** ![GenV](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/genv/32/12132_2.png) [@GenV](https://discuss.huggingface.co/u/GenV)\
**Post date:** [February 28, 2022, 5:01pm UTC](https://discuss.huggingface.co/t/help-for-inference-py-code/15143/5 "2022-02-28T17:01:24Z")

</div>

@philschmid There is a difference in latency because I have two different GPUs in local / remote mode, but it is not significant (it is very low). Using the same versions I have differences using the pipeline () method. I have also a difference in using a .pt or .bin model. I have now switched to the .bin model because I have fewer errors than .pt.

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [March 1, 2022, 1:26pm UTC](https://discuss.huggingface.co/t/help-for-inference-py-code/15143/6 "2022-03-01T13:26:30Z")

</div>

@GenV I might have misunderstood your question. Sorry  
Since you have no switch to the `pytorch_model.bin` you should be able to deploy without the need to create a `inference.py` and just provide the `env` variables when creating the endpoint similar to the snippet below.

```python
from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'t5-base',
	'HF_TASK':'text2text-generation'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.12',
	pytorch_version='1.9',
	py_version='py38',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.g4dn.xlarge' # ec2 instance type
)

predictor.predict({
	'inputs': "Меня зовут Вольфганг и я живу в Берлине"
})

```

You can find more information in the documentation: [Deploy models to Amazon SageMaker](https://huggingface.co/docs/sagemaker/inference)

---

<div class="post-metadata">

**Author:** ![GenV](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/genv/32/12132_2.png) [@GenV](https://discuss.huggingface.co/u/GenV)\
**Post date:** [March 1, 2022, 2:27pm UTC](https://discuss.huggingface.co/t/help-for-inference-py-code/15143/7 "2022-03-01T14:27:24Z")

</div>

@philschmid thank you for the answer. I’m using this code, but with my own model. So my code is:

```auto
role = sagemaker.get_execution_role()

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data=model_data_url, # path to your trained sagemaker model
   role=role, # iam role with permissions to create an Endpoint
   transformers_version=transformers_version, # "4.12.3"
   pytorch_version=pytorch_version, #"1.9.1"
   py_version=py_version # "py38"
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.p3.2xlarge"
)

```

My question is how to create my own inference.py, (or how to implement the model\_fn and transform\_fn methods, because I don’t want to use the pipeline() method but my implementation)

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [March 1, 2022, 2:38pm UTC](https://discuss.huggingface.co/t/help-for-inference-py-code/15143/8 "2022-03-01T14:38:53Z")

</div>

You can take a look here: [Deploy models to Amazon SageMaker](https://huggingface.co/docs/sagemaker/inference#user-defined-code-and-modules)

---

<div class="post-metadata">

**Author:** ![GenV](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/genv/32/12132_2.png) [@GenV](https://discuss.huggingface.co/u/GenV)\
**Post date:** [March 1, 2022, 2:52pm UTC](https://discuss.huggingface.co/t/help-for-inference-py-code/15143/9 "2022-03-01T14:52:27Z")

</div>

Thank you. I don’t see examples/implementations. Are there purely practical examples from which to get ideas for creation?

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [March 7, 2022, 8:03am UTC](https://discuss.huggingface.co/t/help-for-inference-py-code/15143/10 "2022-03-07T08:03:31Z")

</div>

@GenV i created an example on how to do it: [notebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHub](https://github.com/huggingface/notebooks/blob/master/sagemaker/17_custom_inference_script/sagemaker-notebook.ipynb)

---

<div class="post-metadata">

**Author:** ![GenV](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/genv/32/12132_2.png) [@GenV](https://discuss.huggingface.co/u/GenV)\
**Post date:** [March 8, 2022, 9:34am UTC](https://discuss.huggingface.co/t/help-for-inference-py-code/15143/11 "2022-03-08T09:34:57Z")

</div>

@philschmid Thank you! It is very useful.