# Inference Hyperparameters

**URL:** <https://discuss.huggingface.co/t/inference-hyperparameters/9539>\
**Category:** Amazon SageMaker\
**Created:** [August 27, 2021, 7:45pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539 "2021-08-27T19:45:26Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ujjirox](https://avatars.discourse-cdn.com/v4/letter/u/ccd318/32.png) [@ujjirox](https://discuss.huggingface.co/u/ujjirox)\
**Post date:** [August 27, 2021, 7:45pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/1 "2021-08-27T19:45:26Z")

</div>

Hi,

I am interested in deploying a HuggingFace Model on AWS SageMaker. Let’s say for example I deploy “google/pegasus-large” on AWS. You have very generously given the code to deploy this shown below. I was wondering if, as part of the predict function we have additional arguments. I would like to incorporate a custom length penalty as well as repetition penalty. Would you be able to share where in this code these arguments would be inserted?

```auto
from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'google/pegasus-large',
	'HF_TASK':'summarization'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

predictor.predict({
	'inputs': "The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."
})

```

This is what the inference looks like right now in my notebook:

```auto
batch = tokenizer([source], truncation=True, padding='longest', return_tensors="pt").to('cuda')
predicted = tokenizer.batch_decode(model.generate(**batch, length_penalty = 1.5, repetition_penalty=4.), skip_special_tokens=True, )[0]

```

Thanks!

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [August 30, 2021, 7:26am UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/2 "2021-08-30T07:26:39Z")

</div>

Hey @ujjirox,

Nice to hear that you want to use Amazon SageMaker for deploying your Summarization model, and yes it is possible.  
The Inference Toolkit supports the functionalities as the transformers `pipelines`. You can provide all of your additional prediction `parameters` in the `parameters` attribute. I attached an example below of how the request body looks like.

```json
{
	"inputs": "Hugging Face, the winner of VentureBeat’s Innovation in Natural Language Process/Understanding Award for 2021, is looking to level the playing field. The team, launched by Clément Delangue and Julien Chaumond in 2016, was recognized for its work in democratizing NLP, the global market value for which is expected to hit $35.1 billion by 2026. This week, Google’s former head of Ethical AI Margaret Mitchell joined the team.",
	"parameters": {
		"repetition_penalty": 4.0,
		"length_penalty": 1.5
	}
}

```

You can find more information/parameter at [Pipelines — transformers 4.10.1 documentation](https://huggingface.co/transformers/main_classes/pipelines.html) or at our sagemaker documentation [Deploy models to Amazon SageMaker](https://huggingface.co/docs/sagemaker/inference#inference-toolkit---api-description)

Here is an end-to-end example for the `google/pegasus-large` how to deploy and use your parameters.  
_Note: the code snippet is scrollable_

```python
from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'google/pegasus-large',
	'HF_TASK':'summarization'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

long_text= """
Hugging Face, the winner of VentureBeat’s Innovation in Natural Language Process/Understanding Award for 2021, 
is looking to level the playing field. The team, launched by Clément Delangue and Julien Chaumond in 2016,
was recognized for its work in democratizing NLP, the global market value for which is expected to 
hit $35.1 billion by 2026. This week, Google’s former head of Ethical AI Margaret Mitchell joined the team.
"""

parameters = {'repetition_penalty':4.,'length_penalty': 1.5}

predictor.predict({"inputs":long_text,"parameters":parameters})

```

---

<div class="post-metadata">

**Author:** ![ujjirox](https://avatars.discourse-cdn.com/v4/letter/u/ccd318/32.png) [@ujjirox](https://discuss.huggingface.co/u/ujjirox)\
**Post date:** [August 30, 2021, 3:45pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/3 "2021-08-30T15:45:01Z")

</div>

Awesome! Thanks a lot! This is super helpful. Cheers!

---

<div class="post-metadata">

**Author:** ![ujjirox](https://avatars.discourse-cdn.com/v4/letter/u/ccd318/32.png) [@ujjirox](https://discuss.huggingface.co/u/ujjirox)\
**Post date:** [September 2, 2021, 4:53am UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/4 "2021-09-02T04:53:50Z")

</div>

@philschmid

Hey Phil,

Quick question if you don’t mind - are there any arguments in the deployment script to input tags. I am actually getting hit up by my cloud governance team that I need to specify the tags, otherwise it triggers an alert. By tags, I am referring to - [Manage tags - AWS Resource Groups and Tags](https://docs.aws.amazon.com/ARG/latest/userguide/tagging-resources.html)

Thanks!

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 2, 2021, 6:40am UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/5 "2021-09-02T06:40:54Z")

</div>

Hey @ujjirox,

Yes, you can add tags to all related “services” for inference. Meaning in the console:

- Models
- Endpoint Configurations
- Endpoints

To add these tags you need to add a `tags` key to the `.deploy` method, which is a list out of dictionaries with `Key`, `Value` pairs. See below for an example

```python
# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge', # ec2 instance type
    tags=[{'Key':'STAGE', 'Value':'Production'},
          {'Key':'MAINTAINER','Value':'ujjrox'}]
)

```

This will add the tags to all 3 resources

 ![Bildschirmfoto 2021-09-02 um 08.37.36](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/73b6d6f3915e2c428cc7e9363f13a858dd7c7b4b.png)

---

<div class="post-metadata">

**Author:** ![ujjirox](https://avatars.discourse-cdn.com/v4/letter/u/ccd318/32.png) [@ujjirox](https://discuss.huggingface.co/u/ujjirox)\
**Post date:** [September 2, 2021, 1:54pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/6 "2021-09-02T13:54:55Z")

</div>

@philschmid Thank you very much! I’ll leave you in peace!

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 2, 2021, 2:22pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/7 "2021-09-02T14:22:52Z")

</div>

Your welcome! Don’t hesitate to ask if you have further questions or if your company would like to have better support! [Expert Acceleration Program – Hugging Face](https://huggingface.co/support)

---

<div class="post-metadata">

**Author:** ![Oigres](https://avatars.discourse-cdn.com/v4/letter/o/a8b319/32.png) [@Oigres](https://discuss.huggingface.co/u/Oigres)\
**Post date:** [September 15, 2021, 12:32pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/8 "2021-09-15T12:32:04Z")

</div>

Hi.

I’ve been stuck here for a while and I can’t clarify myself.

How do you know what hyperparameters can be used in the pipeline? I mean:

```auto
parameters = {
                  'repetition_penalty': 4.0,
                  'length_penalty': 1.5
}

input_pipeline = {
                  "inputs":long_text,
                  "parameters":parameters
}

```

what other parameters are accepted in “parameters”?

I tried to look for at [SummarizationPipeline class](https://huggingface.co/transformers/main_classes/pipelines.html#transformers.SummarizationPipeline), but I don’t find such parameters, so here is where I stuck… I suppose that it will also depend of the exact model you are use, here for example: `google/pegasus-large`. So my question is:

For a particular model, how to get the exact list of accepted hyparparameters to be used in the pipeline?

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 15, 2021, 1:06pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/9 "2021-09-15T13:06:46Z")

</div>

Hello,

Thanks for the Question. To find all parameters (named) you need to dig a bit deeper into the code. For example, for the `SummarizationPipeline` the [generate](https://huggingface.co/transformers/main_classes/model.html#transformers.generation_utils.GenerationMixin.generate) method is used under the hood. There you can find all available `kwargs`.

I opened a thread internally to improve documentation on this.

---

<div class="post-metadata">

**Author:** ![Oigres](https://avatars.discourse-cdn.com/v4/letter/o/a8b319/32.png) [@Oigres](https://discuss.huggingface.co/u/Oigres)\
**Post date:** [September 15, 2021, 1:35pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/10 "2021-09-15T13:35:31Z")

</div>

Okay.

[Here](https://discuss.huggingface.co/t/how-are-the-inputs-tokenized-when-model-deployment/9692/6) you suggested to me to add the `Truncation` parameter. This is a parameter related with the tokenizer. Why it can be used as “pipeline parameter”. I’m looking for a solution like:

```python
parameters = {
                  'truncation': True,
                  'max_length': 256,
                  'padding': True,
}

```

But I don’t know if the names are right or they aren’t (except `Truncation`, since you suggested the name). Seems that “max\_length” parameter has not any effect on predictions because such predictions are exactly the same with different values.

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 15, 2021, 3:26pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/11 "2021-09-15T15:26:57Z")

</div>

Yes,  
The Parameter would be correct, but the current released `transformers` only support `padding`, and `truncation` as `**kwargs` if you take a look here:

> <https://github.com/huggingface/transformers/blob/a5fc34437dc2cfbd9e91f732820304c017b5525d/src/transformers/pipelines/base.py#L742>

But the next transformers version will allow all `**tokenizers_kwargs` to be passed in.

---

<div class="post-metadata">

**Author:** ![Oigres](https://avatars.discourse-cdn.com/v4/letter/o/a8b319/32.png) [@Oigres](https://discuss.huggingface.co/u/Oigres)\
**Post date:** [September 15, 2021, 4:29pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/12 "2021-09-15T16:29:51Z")

</div>

Ops! This is very bad news :(.

I have a model for classification trained on texts truncated on 256 tokens and I need to deploy it. What would you recommend me to do in such case?.

Greetings!

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 15, 2021, 4:48pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/13 "2021-09-15T16:48:31Z")

</div>

You don’t need to provide the parameters to run inference. It should work like a charm. Padding is not required for inference.  
But if you still want to set the `max_length` to 256 you can modify the tokenizer\_config to 256, example: [tokenizer\_config.json · distilbert-base-uncased-finetuned-sst-2-english at main](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english/blob/main/tokenizer_config.json)

---

<div class="post-metadata">

**Author:** ![Oigres](https://avatars.discourse-cdn.com/v4/letter/o/a8b319/32.png) [@Oigres](https://discuss.huggingface.co/u/Oigres)\
**Post date:** [September 15, 2021, 5:21pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/14 "2021-09-15T17:21:28Z")

</div>

I don’t understand well…

why do you say I don’t need to provide the parameters on inference step? Does a model trained with texts truncated at 256 tokens perform good predictions on texts with more than 256 tokens?

On other hand, where should I put this [tokenizer\_config](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english/blob/main/tokenizer_config.json)…?

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 16, 2021, 6:15am UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/15 "2021-09-16T06:15:16Z")

</div>

Not providing them for Inference will definitely increase the inference speed, since the input won’t be padded and be used as it is.  
@Oigres since the underlying original model was properly pre-trained on 512 tokens it should perform decently well Yes.  
You can add it into your `model.tar.gz` but I would do some tests in advance before deploying.

---

<div class="post-metadata">

**Author:** ![Oigres](https://avatars.discourse-cdn.com/v4/letter/o/a8b319/32.png) [@Oigres](https://discuss.huggingface.co/u/Oigres)\
**Post date:** [September 16, 2021, 8:14am UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/16 "2021-09-16T08:14:46Z")

</div>

Okay.

Finally I retrained the model with 512 tokens and will deploy as it.

Thank you so much.

---

<div class="post-metadata">

**Author:** ![fiona](https://avatars.discourse-cdn.com/v4/letter/f/ecccb3/32.png) [@fiona](https://discuss.huggingface.co/u/fiona)\
**Post date:** [September 24, 2021, 12:23pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/17 "2021-09-24T12:23:54Z")

</div>

My problem is slightly different. I need to invoke the model through a Sagemaker endpoint. I tried to add the parameter and it didn’t work.

client = boto3.client(‘sagemaker-runtime’)  
s = {“inputs”: input\_sentence, “parameters”: {“truncation”:True}}  
payload = json.dumps(s).encode(‘utf-8’)  
content\_type = “application/json”

endpoint\_name = “huggingface-pytorch-inference-2021-09-21-18-08-15-185”  
accept = “application/json”

response = client.invoke\_endpoint(  
EndpointName=endpoint\_name,  
ContentType=content\_type,  
Accept=accept,  
Body=payload,  
)

I’m getting the following error with long input -  
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from model with message "{  
“code”: 400,  
“type”: “InternalServerException”,  
“message”: “The size of tensor a (577) must match the size of tensor b (512) at non-singleton dimension 1”  
}

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 24, 2021, 1:23pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/18 "2021-09-24T13:23:02Z")

</div>

Hello @fiona,

could please share more information like which model you have deployed, how you have created your endpoint etc.

---

<div class="post-metadata">

**Author:** ![fiona](https://avatars.discourse-cdn.com/v4/letter/f/ecccb3/32.png) [@fiona](https://discuss.huggingface.co/u/fiona)\
**Post date:** [September 24, 2021, 2:04pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/19 "2021-09-24T14:04:17Z")

</div>

We have retrained and fine-tuned a Albert xx large classification model. We used the huggingface sagemaker to deploy the model. It works. However we got the above error with a long text.

Here is the deployment code.

task\_env = {  
‘TASK’:‘text-classification’,  
‘HF\_TASK’:‘text-classification’ # NLP task you want to use for predictions  
}

# create Hugging Face Model Class

huggingface\_model = HuggingFaceModel(  
env=task\_env,  
model\_data=saved\_model,  
enable\_network\_isolation=True,  
transformers\_version=“4.6”, # transformers version used  
pytorch\_version=“1.7”, # pytorch version used  
py\_version=“py36”, # python version of the DLC  
)

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 24, 2021, 2:40pm UTC](https://discuss.huggingface.co/t/inference-hyperparameters/9539/20 "2021-09-24T14:40:03Z")

</div>

For me it is working I deployed a

```python
from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'textattack/albert-base-v2-imdb',
	'HF_TASK':'text-classification'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

```

here is how invoked it using `boto3`

```python
import boto3
import json
client = boto3.client('sagemaker-runtime')
request = {
	'inputs': super_long_intput,
    "parameters": {'truncation': True}
}

response = client.invoke_endpoint(
EndpointName="huggingface-pytorch-inference-2021-09-24-14-26-23-337",
ContentType="application/json",
Accept="application/json",
Body=json.dumps(request),
)
response['Body'].read().decode()

```

and a screenshot of that without `{'truncation': True}` i get and error

 ![Bildschirmfoto 2021-09-24 um 16.38.27](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/1a66810fd916d05156df124b9ac78f64b012af4e.png)

[Next page](https://discuss.huggingface.co/t/inference-hyperparameters/9539.md?page=2)