# Use my finetuned Bert Model in SageMaker BatchTransform **URL:** \ **Category:** Amazon SageMaker\ **Created:** [April 28, 2022, 8:40pm UTC](https://discuss.huggingface.co/t/use-my-finetuned-bert-model-in-sagemaker-batchtransform/17312 "2022-04-28T20:40:01Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![bluePenguin](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@bluePenguin](https://discuss.huggingface.co/u/bluePenguin)\ **Post date:** [April 28, 2022, 8:40pm UTC](https://discuss.huggingface.co/t/use-my-finetuned-bert-model-in-sagemaker-batchtransform/17312/1 "2022-04-28T20:40:01Z")
Hi, I finetuned a BERT model on AWS EC2 (i.e. not on SageMaker), and would like to use the resulting model in a Sagemaker Pipeline that ultimately does a BatchTransform step. I’ve saved the model as a pt file and added it to a tar.gz archive, but get an error when I try to use it for inference. I’ve tried adding a json.config file from sample HuggingFace model repos on the HuggingFace website, but still get the same error. I’ve also tried following tutorials to train/finetune a HuggingFace model on Sagemaker, but unfortunately do not have quota for GPU machines for SageMaker training - only for EC2 training off of SageMaker. ```auto from sagemaker.huggingface import HuggingFaceModel import sagemaker role = sagemaker.get_execution_role() # create Hugging Face Model Class huggingface_model = HuggingFaceModel( model_data="https://sagemaker-studio-....s3.amazonaws.com/awsmodel2.tar.gz", # path to your trained sagemaker model role=role, # iam role with permissions to create an Endpoint transformers_version="4.12.3", # transformers version used pytorch_version="1.9.1", # pytorch version used py_version="py38", env={ 'HF_TASK':'text-classification' }# python version of the DLC ) predictor = huggingface_model.deploy( initial_instance_count=1, instance_type="ml.m5.xlarge" ) ``` Error: ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{ “code”: 400, “type”: “InternalServerException”, “message”: “Can\u0027t load config for \u0027/.sagemaker/mms/models/model\u0027. Make sure that:\n\n- \u0027/.sagemaker/mms/models/model\u0027 is a correct model identifier listed on \u0027https://huggingface.co/models\u0027\n (make sure \u0027/.sagemaker/mms/models/model\u0027 is not a path to a local directory with something else, in that case)\n\n- or \u0027/.sagemaker/mms/models/model\u0027 is the correct path to a directory containing a config.json file\n\n” } ". See [https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/huggingface-pytorch-inference-2022-04-28-20-10-46-917](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/huggingface-pytorch-inference-2022-04-28-20-10-46-917) in account XX for more information. ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [April 29, 2022, 1:55pm UTC](https://discuss.huggingface.co/t/use-my-finetuned-bert-model-in-sagemaker-batchtransform/17312/2 "2022-04-29T13:55:41Z")
Hi Rony, it sounds like the model.tar.gz file hasn’t been created properly. Please first make sure that you have all the required files in the tar.gz file, see here: [Deploy models to Amazon SageMaker](https://huggingface.co/docs/sagemaker/inference#create-a-model-artifact-for-deployment) One question also regarding the `config.json` file - you say you used a sample file from the model hub, is that correct? You should use the config file associated with your model that you trained. Hope that helps! Cheers Heiko ---
**Author:** ![bluePenguin](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@bluePenguin](https://discuss.huggingface.co/u/bluePenguin)\ **Post date:** [April 29, 2022, 8:15pm UTC](https://discuss.huggingface.co/t/use-my-finetuned-bert-model-in-sagemaker-batchtransform/17312/3 "2022-04-29T20:15:42Z")
Thank you @marshmellow77! I used ```auto model.save_pretrained("dir_name") after training a model and putting it in eval mode ``` The command created two files: config.json and pytorch\_model.bin I compressed the directory using the following command ```auto tar -czf model_try_429.tar.gz outputs_429/ ``` and uploaded the tar.gz file to S3 Used the same code to create the model as my first post, and got an error when predicting with ```auto data = { "inputs": "The new Hugging Face SageMaker DLC makes it super easy to deploy models in production. I love it!", } predictor.predict(data) ``` > 2022-04-29T20:09:05,830 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last): > 2022-04-29T20:09:05,831 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.8/site-packages/transformers/configuration\_utils.py”, line 594, in \_get\_config\_dict > 2022-04-29T20:09:05,831 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - resolved\_config\_file = cached\_path( > 2022-04-29T20:09:05,831 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.8/site-packages/transformers/file\_utils.py”, line 1936, in cached\_path > 2022-04-29T20:09:05,831 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise EnvironmentError(f"file {url\_or\_filename} not found") > 2022-04-29T20:09:05,831 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - OSError: file /.sagemaker/mms/models/model/config.json not found > 2022-04-29T20:09:05,831 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - > 2022-04-29T20:09:05,832 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred: > 2022-04-29T20:09:05,832 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - > 2022-04-29T20:09:05,832 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last): > 2022-04-29T20:09:05,832 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.8/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”, line 219, in handle > 2022-04-29T20:09:05,832 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.initialize(context) > 2022-04-29T20:09:05,832 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.8/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”, line 77, in initialize > 2022-04-29T20:09:05,833 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.model = self.load(self.model\_dir) > 2022-04-29T20:09:05,833 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.8/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”, line 104, in load > 2022-04-29T20:09:05,833 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf\_pipeline = get\_pipeline(task=os.environ[“HF\_TASK”], model\_dir=model\_dir, device=self.device) > 2022-04-29T20:09:05,833 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.8/site-packages/sagemaker\_huggingface\_inference\_toolkit/transformers\_utils.py”, line 272, in get\_pipeline > 2022-04-29T20:09:05,834 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf\_pipeline = pipeline(task=task, model=model\_dir, device=device, \*\*kwargs) > 2022-04-29T20:09:05,834 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.8/site-packages/transformers/pipelines/ **init**.py”, line 541, in pipeline > 2022-04-29T20:09:05,834 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - config = AutoConfig.from\_pretrained(model, revision=revision, \_from\_pipeline=task, \*\*model\_kwargs) > 2022-04-29T20:09:05,835 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.8/site-packages/transformers/models/auto/configuration\_auto.py”, line 637, in from\_pretrained > 2022-04-29T20:09:05,835 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - config\_dict, \_ = PretrainedConfig.get\_config\_dict(pretrained\_model\_name\_or\_path, \*\*kwargs) > 2022-04-29T20:09:05,835 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.8/site-packages/transformers/configuration\_utils.py”, line 546, in get\_config\_dict > 2022-04-29T20:09:05,836 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - config\_dict, kwargs = cls.\_get\_config\_dict(pretrained\_model\_name\_or\_path, \*\*kwargs) > 2022-04-29T20:09:05,836 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.8/site-packages/transformers/configuration\_utils.py”, line 630, in \_get\_config\_dict > 2022-04-29T20:09:05,836 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise EnvironmentError( > 2022-04-29T20:09:05,837 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - OSError: Can’t load config for ‘/.sagemaker/mms/models/model’. If you were trying to load it from ‘[Models - Hugging Face](https://huggingface.co/models)’, make sure you don’t have a local directory with the same name. Otherwise, make sure ‘/.sagemaker/mms/models/model’ is the correct path to a directory containing a config.json file > 2022-04-29T20:09:05,837 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - > 2022-04-29T20:09:05,837 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred: > 2022-04-29T20:09:05,837 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - > 2022-04-29T20:09:05,838 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last): > 2022-04-29T20:09:05,841 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.8/site-packages/mms/service.py”, line 108, in predict > 2022-04-29T20:09:05,844 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ret = self.\_entry\_point(input\_batch, self.context) > 2022-04-29T20:09:05,845 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.8/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”, line 243, in handle > 2022-04-29T20:09:05,845 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise PredictionException(str(e), 400) > 2022-04-29T20:09:05,845 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: Can’t load config for ‘/.sagemaker/mms/models/model’. If you were trying to load it from ‘[Models - Hugging Face](https://huggingface.co/models)’, make sure you don’t have a local directory with the same name. Otherwise, make sure ‘/.sagemaker/mms/models/model’ is the correct path to a directory containing a config.json file : 400 ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [April 29, 2022, 9:05pm UTC](https://discuss.huggingface.co/t/use-my-finetuned-bert-model-in-sagemaker-batchtransform/17312/4 "2022-04-29T21:05:27Z")
> [@bluePenguin](#): > > ` tar -czf model_try_429.tar.gz outputs_429/` I suspect that this `tar` command is incorrect: 1. You seem to zip the directory instead of the files, which would mean that the structure of your file will be incorrect. 2. The file name HAS to be `model.tar.gz`, as far as I know. Instead you could try to navigate into the directory and use `tar zcvf model.tar.gz *`, as described in the [documentation](https://huggingface.co/docs/sagemaker/inference#create-a-model-artifact-for-deployment): ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/4/451edb915b79c007ff565264fcbeb262f7b3a22e.png) In the end you want the `model.tar.gz` structure look like this: ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/9/94fb0e64406e8ad2afc1112d0f31c6347475b5ac.png) ---
**Author:** ![bluePenguin](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@bluePenguin](https://discuss.huggingface.co/u/bluePenguin)\ **Post date:** [April 30, 2022, 12:57am UTC](https://discuss.huggingface.co/t/use-my-finetuned-bert-model-in-sagemaker-batchtransform/17312/5 "2022-04-30T00:57:24Z")
@marshmellow77 That fixed it, thank you so much!