# Save and deploy distilbert model in AWS SageMaker

**URL:** <https://discuss.huggingface.co/t/save-and-deploy-distilbert-model-in-aws-sagemaker/5329>\
**Category:** 🤗Transformers\
**Created:** [April 6, 2021, 1:23pm UTC](https://discuss.huggingface.co/t/save-and-deploy-distilbert-model-in-aws-sagemaker/5329 "2021-04-06T13:23:13Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![gopalkr272](https://avatars.discourse-cdn.com/v4/letter/g/fbc32d/32.png) [@gopalkr272](https://discuss.huggingface.co/u/gopalkr272)\
**Post date:** [April 6, 2021, 1:23pm UTC](https://discuss.huggingface.co/t/save-and-deploy-distilbert-model-in-aws-sagemaker/5329/1 "2021-04-06T13:23:13Z")

</div>

I am trying to download the Hugging Face distilbert model, trying to save to S3. The model itself does not have a deploy method. So I am saving to S3, instantiating it and trying to deploy. May I know if this will work with Sagemaker. What I am doing wrong. Here are the steps:

model\_name = ‘distilbert-base-uncased-distilled-squad’ model = DistilBertForQuestionAnswering.from\_pretrained(model\_name) tokenizer = DistilBertTokenizerFast.from\_pretrained(model\_name)

the below works and gives output context = “xxxx” question = “yyy?”

nlp = pipeline(‘question-answering’, model=model, tokenizer=tokenizer)

nlp({ ‘question’: ‘What organization is the IPCC a part of?’, ‘context’: context })

save the model to local folder model.save\_pretrained(’./scripts/mymodel’)

zip the model file. with tarfile.open("./scripts/mymodel/model.tar.gz", “w:gz”) as tar: tar.add("./scripts/mymodel/pytorch\_model.bin") tar.add("./scripts/mymodel/config.json")

upload the zipped file to S3 sagemaker.Session().upload\_data(bucket=sagemaker\_session\_bucket, path=’./scripts/mymodel/model.tar.gz’, key\_prefix=‘model’)

instantiating the saved model

bertmodel = PyTorchModel(entry\_point=‘inference.py’, source\_dir=‘scripts’, model\_data=‘s3://’+sagemaker\_session\_bucket+’/model/model.tar.gz’, role=sagemaker.get\_execution\_role(), framework\_version=‘1.5’, py\_version=‘py3’)

the below does not work nlp = pipeline(‘question-answering’, model=bertmodel, tokenizer=tokenizer)

nlp({ ‘question’: ‘What organization is the IPCC a part of?’, ‘context’: context })

error recd - AttributeError: ‘PyTorchModel’ object has no attribute ‘config’

I am able to deploy the predictor predictor = bertmodel.deploy(initial\_instance\_count=1, instance\_type=‘ml.m5.xlarge’)

---

<div class="post-metadata">

**Author:** ![g3casey](https://avatars.discourse-cdn.com/v4/letter/g/e19b73/32.png) [@g3casey](https://discuss.huggingface.co/u/g3casey)\
**Post date:** [April 8, 2021, 2:47am UTC](https://discuss.huggingface.co/t/save-and-deploy-distilbert-model-in-aws-sagemaker/5329/2 "2021-04-08T02:47:05Z")

</div>

@gopalkr272 I am afraid I don’t have a complete answer but review my support ticket here:  
[https://github.com/huggingface/transformers/issues/11043](https://github.com/huggingface/transformers/issues/11043)  
I hope this will help you find an answer.

---

<div class="post-metadata">

**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\
**Post date:** [April 9, 2021, 8:06am UTC](https://discuss.huggingface.co/t/save-and-deploy-distilbert-model-in-aws-sagemaker/5329/3 "2021-04-09T08:06:17Z")

</div>

There are couple public samples showing Hugging Face model deployment on SageMaker, for example those:

- [amazon-sagemaker-deploy-nlp-huggingface/deploy\_pretrained\_BART\_seq2seq\_PyTorch.ipynb at main · aws-samples/amazon-sagemaker-deploy-nlp-huggingface · GitHub](https://github.com/aws-samples/amazon-sagemaker-deploy-nlp-huggingface/blob/main/summarizer/single-model/deploy_pretrained_BART_seq2seq_PyTorch.ipynb)
- [Serving PyTorch models in production with the Amazon SageMaker native TorchServe integration | AWS Machine Learning Blog](https://aws.amazon.com/blogs/machine-learning/serving-pytorch-models-in-production-with-the-amazon-sagemaker-native-torchserve-integration/)
- [GitHub - aws-samples/amazon-sagemaker-bert-classify-pytorch: This sample show you how to train BERT on Amazon Sagemaker using Spot instances](https://github.com/aws-samples/amazon-sagemaker-bert-classify-pytorch)