# Slow inference using most recent docker image

**URL:** <https://discuss.huggingface.co/t/slow-inference-using-most-recent-docker-image/12423>\
**Category:** Amazon SageMaker\
**Created:** [December 1, 2021, 4:25pm UTC](https://discuss.huggingface.co/t/slow-inference-using-most-recent-docker-image/12423 "2021-12-01T16:25:39Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ojturner](https://avatars.discourse-cdn.com/v4/letter/o/ee7513/32.png) [@ojturner](https://discuss.huggingface.co/u/ojturner)\
**Post date:** [December 1, 2021, 4:25pm UTC](https://discuss.huggingface.co/t/slow-inference-using-most-recent-docker-image/12423/1 "2021-12-01T16:25:39Z")

</div>

We have been deploying a BERT model to a SageMaker endpoint using the g4dn.xlarge instance. The deploy is managed with terraform and we use the following image:

```auto
{account}.dkr.ecr.{region}.amazonaws.com/huggingface-pytorch-inference:1.7-transformers4.6-gpu-py36-cu110-ubuntu18.04

```

We’re really happy with the model latency, which is about 0.05 seconds for the typical request.

Now we were experimenting with using the more recent image:

```auto
{account}.dkr.ecr.{region}.amazonaws.com/huggingface-pytorch-inference:1.9.1-transformers4.12.3-gpu-py38-cu111-ubuntu20.04 

```

and found that model latency dropped by a factor of about 10, everything else being equal. I noticed in the sagemaker sdk docs that it says the supported versions of pytorch, transformers and python respectively are 1.7.1, 4.6.1 and 3.6 respectively.  
So my question is more of a curiosity - why does the model latency drop on this most recent image? Is there some issue using the GPU here?  
Thanks,  
Owen

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [December 2, 2021, 7:12am UTC](https://discuss.huggingface.co/t/slow-inference-using-most-recent-docker-image/12423/2 "2021-12-02T07:12:15Z")

</div>

Hey @ojturner,

Thank you for opening the Thread. Performance drop should not appear! Can you share which model (at least the architecture and task) you use and which instance type?

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [December 2, 2021, 8:03am UTC](https://discuss.huggingface.co/t/slow-inference-using-most-recent-docker-image/12423/3 "2021-12-02T08:03:32Z")

</div>

@ojturner i tested it myself using the [distilbert/distilbert-base-uncased-finetuned-sst-2-english · Hugging Face](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english) model with one time `transformers=4.6.1` and one time `transformers=4.12.3`.  
For me `4.12.3` is achieving better results than `4.6.1`

### `4.6.1`

 ![Bildschirmfoto 2021-12-02 um 09.00.46](https://us1.discourse-cdn.com/hellohellohello/original/2X/4/48d41566502280722a371608ef86dba05a543e1a.png)

### `4.12.3`

 ![Bildschirmfoto 2021-12-02 um 09.03.08](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/d41b728a1eeaff6910e6a192d98fb2e0544d3431.png)

---

<div class="post-metadata">

**Author:** ![ojturner](https://avatars.discourse-cdn.com/v4/letter/o/ee7513/32.png) [@ojturner](https://discuss.huggingface.co/u/ojturner)\
**Post date:** [December 2, 2021, 11:41am UTC](https://discuss.huggingface.co/t/slow-inference-using-most-recent-docker-image/12423/4 "2021-12-02T11:41:00Z")

</div>

Hey @philschmid thanks for your response and for looking into the model performance. I am using the distilbert-base-uncased model, finetuned using a custom dataset for multiclass categorisation. The task is text-classification and the instance used for deployment is ml.g4dn.xlarge.

I tested this morning deploying with the sagemaker sdk, instead of terraform, and found the same issue. For the first deployment I did the following:

```auto
ENDPOINT_NAME = "bert-latency-test"
huggingface_model = HuggingFaceModel(
    role="-",
    model_data="-",
    entry_point="-",
    source_dir="-",
    code_location="-",
    transformers_version='4.6',
    pytorch_version='1.7',
    py_version='py36',
)
pred = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type="ml.g4dn.xlarge",
    endpoint_name=ENDPOINT_NAME,
)

```

and made requests with

```auto
request = {"inputs": [-]}
pred.predict(request)

```

The “inputs” in each request is a list of 25 short strings. The image used for the container associated with the created SageMaker model is:

```auto
-/huggingface-pytorch-inference:1.7-transformers4.6-gpu-py36-cu110-ubuntu18.04

```

attaching a similar screenshot from cloudwatch showing model and overhead latency

 ![Screenshot 2021-12-02 at 10.46.00](https://us1.discourse-cdn.com/hellohellohello/original/2X/e/e2cbecc783bbe1363a1ab547227c1716eaa5c88f.png)

For the second test everything stays constant except the transformers, python and pytorch version:

```auto
ENDPOINT_NAME = "bert-latency-test"
huggingface_model = HuggingFaceModel(
    role="-",
    model_data="-",
    entry_point="-",
    source_dir="-",
    code_location="-",
    transformers_version='4.12.3',
    pytorch_version='1.9.1',
    py_version='py38',
)
pred = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type="ml.g4dn.xlarge",
    endpoint_name=ENDPOINT_NAME,
)

```

which creates a mode using the following image:

```auto
-/huggingface-pytorch-inference:1.9.1-transformers4.12.3-gpu-py38-cu111-ubuntu20.04

```

Requests are sent in the same way, and it’s here I observe the drop in model latency as per the following screenshot from cloudwatch.  
SEE BELOW REPLY FOR HIGH MODEL LATENCY SCREENSHOT

Would be very appreciative of any advice you can give here.

On top of this (and this should possibly go in a separate thread) the way our application will make requests to the endpoint is through a sagemaker-runtime client. I’ve noticed there is quite a jump in OverheadLatency when doing this, instead of using the sagemaker Predictor object. e.g.

```auto
import boto3
from sagemaker.serializers import JSONSerializer
client = boto3.client("sagemaker-runtime")
client.invoke_endpoint(EndpointName=ENDPOINT_NAME, Body=JSONSerializer().serialize(request) ContentType="application/json")

```

SEE BELOW REPLY FOR HIGH OVERHEAD LATENCY SCREENSHOT

Is there anything I can do to minimise the overhead latency here?

noticed that new users can only embed one media per post, so I’ll put the other screenshots in other replies.

Many thanks for your help,  
Owen

---

<div class="post-metadata">

**Author:** ![ojturner](https://avatars.discourse-cdn.com/v4/letter/o/ee7513/32.png) [@ojturner](https://discuss.huggingface.co/u/ojturner)\
**Post date:** [December 2, 2021, 11:41am UTC](https://discuss.huggingface.co/t/slow-inference-using-most-recent-docker-image/12423/5 "2021-12-02T11:41:47Z")

</div>

HIGH MODEL LATENCY SCREENSHOT

 ![Screenshot 2021-12-02 at 11.25.11](https://us1.discourse-cdn.com/hellohellohello/original/2X/e/eaa7054a90f0f197830d699e895825e81fe1ef3a.png)

---

<div class="post-metadata">

**Author:** ![ojturner](https://avatars.discourse-cdn.com/v4/letter/o/ee7513/32.png) [@ojturner](https://discuss.huggingface.co/u/ojturner)\
**Post date:** [December 2, 2021, 11:42am UTC](https://discuss.huggingface.co/t/slow-inference-using-most-recent-docker-image/12423/6 "2021-12-02T11:42:18Z")

</div>

HIGH OVERHEAD LATENCY SCREENSHOT

 ![Screenshot 2021-12-02 at 10.52.28](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/76c3f1622e28665d8a6c0517ff442f19a344416a.png)

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [December 7, 2021, 5:44pm UTC](https://discuss.huggingface.co/t/slow-inference-using-most-recent-docker-image/12423/7 "2021-12-07T17:44:39Z")

</div>

Hello @ojturner,

From your code

> [@ojturner](#):
>
> ```auto
> huggingface_model = HuggingFaceModel(
> role="-",
> model_data="-",
> entry_point="-",
> source_dir="-",
> code_location="-",
> transformers_version='4.12.3',
> pytorch_version='1.9.1',
> py_version='py38',
> )
> 
> ```

it looks like you are using a custom `inference.py` script is that correct? could you provide? Have you tested the latency and overhead using the “zero-code” deployment, without providing `inference.py`.

Could you also share more information about which model/model-architecture/task you are using?

---

<div class="post-metadata">

**Author:** ![ojturner](https://avatars.discourse-cdn.com/v4/letter/o/ee7513/32.png) [@ojturner](https://discuss.huggingface.co/u/ojturner)\
**Post date:** [December 9, 2021, 3:55pm UTC](https://discuss.huggingface.co/t/slow-inference-using-most-recent-docker-image/12423/8 "2021-12-09T15:55:58Z")

</div>

Hey @philschmid - yes that’s right, we’ve been using a custom inference.py script. The only thing we have in there is a custom model\_fn to return all scores, instead of just the top category for our categorisation task:

```auto
import os
from sagemaker_huggingface_inference_toolkit import transformers_utils

GPU_ID = 0
GPU_NOT_AVAILABLE_ID = -1

def model_fn(model_dir):
    """
    The Load handler is responsible for loading the Hugging Face transformer model.
    It can be overridden to load the model from storage
    Returns:
        hf_pipeline (Pipeline): A Hugging Face Transformer pipeline.
    """
    config_file = "config.json"
    if not config_file in os.listdir(model_dir):
        raise ValueError(f"{config_file} not found", 403)

    task = transformers_utils.infer_task_from_model_architecture(f"{model_dir}/{config_file}")
    device_id = GPU_ID if transformers_utils._is_gpu_available() else GPU_NOT_AVAILABLE_ID

    hf_pipeline = transformers_utils.get_pipeline(
        task=task,
        model_dir=model_dir,
        device=device_id,
        return_all_scores=True,
    )

    return hf_pipeline

```

But we’ve also tried the “zero-code” deployment and inference is just as slow using transformers 4.12.3.

The task is sequence classification, in this case there are 60 different possible targets. The model is loaded as:

```auto
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=60)

```

and then we fine tune on our labelled dataset. Let me know if you require any more information, thank you. Owen

---

<div class="post-metadata">

**Author:** ![ojturner](https://avatars.discourse-cdn.com/v4/letter/o/ee7513/32.png) [@ojturner](https://discuss.huggingface.co/u/ojturner)\
**Post date:** [January 3, 2022, 10:38am UTC](https://discuss.huggingface.co/t/slow-inference-using-most-recent-docker-image/12423/9 "2022-01-03T10:38:04Z")

</div>

Hey @philschmid happy new year!

Was wondering if you had any more thoughts on this issue?

We found that when we compiled the model for use with one of the inferentia endpoints and wrote our own predict\_fn, we didn’t have any latency issues with the latest code version. Presumably then the original latency problem stems from making predictions using the huggingface pipeline object in the latest code version?

Thank you, Owen

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [January 7, 2022, 8:13am UTC](https://discuss.huggingface.co/t/slow-inference-using-most-recent-docker-image/12423/10 "2022-01-07T08:13:26Z")

</div>

Happy new year as well 🎊

Could potentially be possible. You could confirm this by adding a `requirements.txt` next to your `inference.py` script and adding a new `transformers` version.

Since you are using Inferentia already, I am happy to share that there will be soon HF-specific Inferentia DLCs.

---

<div class="post-metadata">

**Author:** ![ojturner](https://avatars.discourse-cdn.com/v4/letter/o/ee7513/32.png) [@ojturner](https://discuss.huggingface.co/u/ojturner)\
**Post date:** [March 21, 2022, 3:03pm UTC](https://discuss.huggingface.co/t/slow-inference-using-most-recent-docker-image/12423/11 "2022-03-21T15:03:08Z")

</div>

Hey @philschmid just for completeness we figured out the issue here.  
We pinned the latency increase down as occurring in the jump from version 4.10 → 4.11, when a lot of pipeline refactoring was done. As part of this the batch\_size argument was introduced as an argument when creating a pipeline, with a default value of 1. Changing this value to something bigger solved our latency issue.