# Error: Could Not Load Model

**URL:** <https://discuss.huggingface.co/t/error-could-not-load-model/15362>\
**Category:** Amazon SageMaker\
**Created:** [March 4, 2022, 3:51pm UTC](https://discuss.huggingface.co/t/error-could-not-load-model/15362 "2022-03-04T15:51:07Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![dbounds](https://avatars.discourse-cdn.com/v4/letter/d/51bf81/32.png) [@dbounds](https://discuss.huggingface.co/u/dbounds)\
**Post date:** [March 4, 2022, 3:51pm UTC](https://discuss.huggingface.co/t/error-could-not-load-model/15362/1 "2022-03-04T15:51:07Z")

</div>

Working on a project that needs to deploy raw HF models without training them using SageMaker Endpoints. I clone the model repo from the HF repo, tar.gz it, load it onto S3, create my SageMaker Model, endpoint configuration, and deploy my endpoint. When I send a text payload to the endpoint, I get the below error.  
I originally thought I was having the same problem as this thread: [How to Create Model in SageMaker Console from .tar.gz](https://discuss.huggingface.co/t/how-to-create-model-in-sagemaker-console-from-tar-gz/14830)  
but the issue persists even after adding ‘Environment’: {“MMS\_DEFAULT\_WORKERS\_PER\_MODEL”: ‘1’} to my create\_model call.  
Any ideas on what I might be doing wrong?

SageMaker Error:  
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from model with message "{  
“code”: 400,  
“type”: “InternalServerException”,  
“message”: “Could not load model /.sagemaker/mms/models/model with any of the following classes: (\u003cclass \u0027transformers.models.auto.modeling\_auto.AutoModelForMaskedLM\u0027\u003e, \u003cclass \u0027transformers.models.distilbert.modeling\_distilbert.DistilBertForMaskedLM\u0027\u003e).”  
}

Error from CloudWatch for the endpoint:  
python: can’t open file ‘/usr/local/bin/deep\_learning\_container.py’: [Errno 13] Permission denied

My model definition:

huggingface\_model\_config = client.create\_model(  
ModelName = “nlp-serverless-model-” + strftime("%Y-%m-%d-%H-%M-%S", gmtime()),  
ExecutionRoleArn = role,  
Containers = [  
{  
‘Image’: ‘[763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:1.9.1-transformers4.12.3-cpu-py38-ubuntu20.04](http://763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:1.9.1-transformers4.12.3-cpu-py38-ubuntu20.04)’,  
‘Mode’: ‘SingleModel’,  
‘ModelDataUrl’: model\_file,  
‘Environment’: {“MMS\_DEFAULT\_WORKERS\_PER\_MODEL”: ‘1’}  
}  
],  
)

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [March 7, 2022, 8:12am UTC](https://discuss.huggingface.co/t/error-could-not-load-model/15362/2 "2022-03-07T08:12:10Z")

</div>

> [@dbounds](#):
>
> SageMaker Error:  
> ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from model with message "{  
> “code”: 400,  
> “type”: “InternalServerException”,  
> “message”: “Could not load model /.sagemaker/mms/models/model with any of the following classes: (\u003cclass \u0027transformers.models.auto.modeling\_auto.AutoModelForMaskedLM\u0027\u003e, \u003cclass \u0027transformers.models.distilbert.modeling\_distilbert.DistilBertForMaskedLM\u0027\u003e).”  
> }

Mostly means you screwed up the archive/creation of the `model.tar.gz`. You can find documentation and instructions here: [Deploy models to Amazon SageMaker](https://huggingface.co/docs/sagemaker/inference#create-a-model-artifact-for-deployment).

Are you using an `inference.py` script or the `env` config with `HF_TASK`? Could you also please share the code for `endpoint-configuration` and `endpoint`?

---

<div class="post-metadata">

**Author:** ![dbb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dbb/32/19757_2.png) [@dbb](https://discuss.huggingface.co/u/dbb)\
**Post date:** [March 10, 2022, 11:08am UTC](https://discuss.huggingface.co/t/error-could-not-load-model/15362/3 "2022-03-10T11:08:55Z")

</div>

@dbounds , I had a call with AWS support a few minutes ago and we rebuild the whole config via the console. I created a little manual here:

> [@How to Create Model in SageMaker Console from .tar.gz](https://discuss.huggingface.co/t/how-to-create-model-in-sagemaker-console-from-tar-gz/14830/8):
>
> @philschmid @YannAgora , thanks for your help! I just want to give you a short update regarding the case and maybe the following is also relevant for everyone who encounters the problem or wants to set up serverless inference via the SageMaker console slight_smile TL;DR Endpoint returns prediction BUT error still visible in logs python: can't open file '/usr/local/bin/deep\_learning\_container.py': [Errno 13] Permission denied Re-build the endpoint configuration with max. ram (6GB) magically…

maybe you can give it a try?

Best,  
David

---

<div class="post-metadata">

**Author:** ![dbounds](https://avatars.discourse-cdn.com/v4/letter/d/51bf81/32.png) [@dbounds](https://discuss.huggingface.co/u/dbounds)\
**Post date:** [March 11, 2022, 1:15pm UTC](https://discuss.huggingface.co/t/error-could-not-load-model/15362/4 "2022-03-11T13:15:47Z")

</div>

Thank you for the guidance, tried the directions in the link with the same result.

I am doing a git checkout of the specified model, distilber-base-uncased in this case, creating a model.tar.gx and loading that into s3 as the target model (for this use case, I am skipping training)

Model, endpoint config and endpoint code blocks below:

```auto
huggingface_model = HuggingFaceModel(
    model_data=f"s3://{s3_bucket}/{s3_prefix}/model.tar.gz",
    role=role,
    transformers_version="4.12.3",
    pytorch_version="1.9.1",
    py_version='py38',
    env={
        'HF_TASK': 'text-classification'
    },
)

```

```auto
huggingface_model_config = client.create_model(
    ModelName = "nlp-serverless-model-" + strftime("%Y-%m-%d-%H-%M-%S", gmtime()),
    ExecutionRoleArn = role,
    Containers = [
        {
            'Image': '763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:1.9.1-transformers4.12.3-cpu-py38-ubuntu20.04',
            'Mode': 'SingleModel',
            'ModelDataUrl': model_file,
            'Environment': {"MMS_DEFAULT_WORKERS_PER_MODEL": '1'}
        }
    ],
)

```

```auto
endpoint_config_response = client.create_endpoint_config(
    EndpointConfigName=epc_name,
    ProductionVariants=[
        {
            'VariantName': 'single-variant',
            'ModelName': sm_model_name.split("/")[1],
            'ServerlessConfig': {
                'MemorySizeInMB': 6144,
                'MaxConcurrency': 10,
            },
        },
    ],
)

```

```auto
create_endpoint_response = client.create_endpoint(
    EndpointName=endpoint_name,
    EndpointConfigName=epc_name,
)

```

---

<div class="post-metadata">

**Author:** ![dbounds](https://avatars.discourse-cdn.com/v4/letter/d/51bf81/32.png) [@dbounds](https://discuss.huggingface.co/u/dbounds)\
**Post date:** [March 11, 2022, 1:17pm UTC](https://discuss.huggingface.co/t/error-could-not-load-model/15362/5 "2022-03-11T13:17:36Z")

</div>

DBB: Appreciate the link. Tired again with 6144 set as MemorySizeInMb with the same result.

---

<div class="post-metadata">

**Author:** ![anwarika](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@anwarika](https://discuss.huggingface.co/u/anwarika)\
**Post date:** [March 11, 2022, 1:50pm UTC](https://discuss.huggingface.co/t/error-could-not-load-model/15362/6 "2022-03-11T13:50:40Z")

</div>

I’ve run into an issue like this before, how’d you tar your file? Are you using a Mac?

---

<div class="post-metadata">

**Author:** ![dbounds](https://avatars.discourse-cdn.com/v4/letter/d/51bf81/32.png) [@dbounds](https://discuss.huggingface.co/u/dbounds)\
**Post date:** [March 11, 2022, 1:52pm UTC](https://discuss.huggingface.co/t/error-could-not-load-model/15362/7 "2022-03-11T13:52:29Z")

</div>

I removed the model objects I did not need (tried without this step as well with the same result) and created the tar from a SageMaker notebook instance:

```auto
!rm flax_model.msgpack
!rm rust_model.ot
!rm tf_model.h5
!tar zcvf model.tar.gz *

```

---

<div class="post-metadata">

**Author:** ![anwarika](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@anwarika](https://discuss.huggingface.co/u/anwarika)\
**Post date:** [March 11, 2022, 1:57pm UTC](https://discuss.huggingface.co/t/error-could-not-load-model/15362/8 "2022-03-11T13:57:55Z")

</div>

In sagemaker I use the `tarfile` package. Have you tried different Image versions? Maybe its a dependency issue