# ModelError when I run predict after deploying wizardcoder for text-generation **URL:** \ **Category:** Amazon SageMaker\ **Created:** [September 22, 2023, 5:10pm UTC](https://discuss.huggingface.co/t/modelerror-when-i-run-predict-after-deploying-wizardcoder-for-text-generation/56032 "2023-09-22T17:10:28Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![HuggingKiranFace](https://avatars.discourse-cdn.com/v4/letter/h/ba8739/32.png) [@HuggingKiranFace](https://discuss.huggingface.co/u/HuggingKiranFace)\ **Post date:** [September 22, 2023, 5:10pm UTC](https://discuss.huggingface.co/t/modelerror-when-i-run-predict-after-deploying-wizardcoder-for-text-generation/56032/1 "2023-09-22T17:10:28Z")
**Here’s my code:** ```auto import json from sagemaker.huggingface import HuggingFaceModel import sagemaker env = {'HF_TASK': 'text-generation'} #create Hugging Face Model Class huggingface_model = HuggingFaceModel( model_data="s3://generative-ai/WizardCoder/model.tar.gz", # path to your trained sagemaker model role=role, # iam role with permissions to create an Endpoint transformers_version="4.26", # Transformers version used env=env, pytorch_version="1.13", # PyTorch version used py_version='py39', # Python version used ) #deploy model to SageMaker Inference predictor = huggingface_model.deploy( initial_instance_count=1, instance_type="ml.g5.2xlarge" ) # send request predictor.predict({ "inputs": "Write a python program to reverse a string", }) ``` **I am getting below error:** * * * ModelError Traceback (most recent call last) Cell In[24], line 2 1 # send request ----\> 2 predictor.predict({ 3 “inputs”: “Write a python program to reverse a string”, 4 }) File ~/anaconda3/envs/tensorflow2\_p310/lib/python3.10/site-packages/sagemaker/base\_predictor.py:185, in Predictor.predict(self, data, initial\_args, target\_model, target\_variant, inference\_id, custom\_attributes) 138 “”“Return the inference from the specified endpoint. 139 140 Args: (…) 174 as is. 175 “”” 177 request\_args = self.\_create\_request\_args( 178 data, 179 initial\_args, (…) 183 custom\_attributes, 184 ) → 185 response = self.sagemaker\_session.sagemaker\_runtime\_client.invoke\_endpoint(\*\*request\_args) 186 return self.\_handle\_response(response) File ~/anaconda3/envs/tensorflow2\_p310/lib/python3.10/site-packages/botocore/client.py:535, in ClientCreator.\_create\_api\_method..\_api\_call(self, \*args, \*\*kwargs) 531 raise TypeError( 532 f"{py\_operation\_name}() only accepts keyword arguments." 533 ) 534 # The “self” in this scope is referring to the BaseClient. → 535 return self.\_make\_api\_call(operation\_name, kwargs) File ~/anaconda3/envs/tensorflow2\_p310/lib/python3.10/site-packages/botocore/client.py:980, in BaseClient.\_make\_api\_call(self, operation\_name, api\_params) 978 error\_code = parsed\_response.get(“Error”, {}).get(“Code”) 979 error\_class = self.exceptions.from\_code(error\_code) → 980 raise error\_class(parsed\_response, operation\_name) 981 else: 982 return parsed\_response ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{ “code”: 400, “type”: “InternalServerException”, “message”: “\u0027llama\u0027” } ". See [https://ap-south-1.console.aws.amazon.com/cloudwatch/home?region=ap-south-1#logEventViewer:group=/aws/sagemaker/Endpoints/huggingface-pytorch-inference-2023-09-22-11-46-53-545](https://ap-south-1.console.aws.amazon.com/cloudwatch/home?region=ap-south-1#logEventViewer:group=/aws/sagemaker/Endpoints/huggingface-pytorch-inference-2023-09-22-11-46-53-545) in account 346264802683 for more information. **Below is the cloud watch logs** Warning: MMS is using non-default JVM parameters: -XX:-UseContainerSupport WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. 2023-09-22T16:51:20,034 [INFO] main com.amazonaws.ml.mms.ModelServer - 2023-09-22T16:51:20,038 [INFO] main com.amazonaws.ml.mms.ModelServer - Loading initial models: /opt/ml/model preload\_model: false 2023-09-22T16:51:20,067 [WARN] W-9000-model com.amazonaws.ml.mms.wlm.WorkerLifeCycle - attachIOStreams() threadName=W-9000-model 2023-09-22T16:51:20,124 [INFO] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model\_service\_worker started with args: --sock-type unix --sock-name /home/model-server/tmp/.mms.sock.9000 --handler sagemaker\_huggingface\_inference\_toolkit.handler\_service --model-path /opt/ml/model --model-name model --preload-model false --tmp-dir /home/model-server/tmp 2023-09-22T16:51:20,126 [INFO] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Listening on port: /home/model-server/tmp/.mms.sock.9000 2023-09-22T16:51:20,126 [INFO] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - [PID] 54 2023-09-22T16:51:20,127 [INFO] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Python runtime: 3.9.13 2023-09-22T16:51:20,128 [INFO] main com.amazonaws.ml.mms.wlm.ModelManager - Model model loaded. 2023-09-22T16:51:20,132 [INFO] main com.amazonaws.ml.mms.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2023-09-22T16:51:20,139 [INFO] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.mms.sock.9000 2023-09-22T16:51:20,186 [INFO] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Connection accepted: /home/model-server/tmp/.mms.sock.9000. Model server started. 2023-09-22T16:51:20,272 [WARN] pool-3-thread-1 com.amazonaws.ml.mms.metrics.MetricCollector - worker pid is not available yet. 2023-09-22T16:51:22,461 [INFO] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ‘llama’ 2023-09-22T16:51:22,464 [INFO] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.model = self.load(self.model\_dir) 2023-09-22T16:51:22,465 [INFO] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf\_pipeline = get\_pipeline(task=os.environ[“HF\_TASK”], model\_dir=model\_dir, device=self.device) 2023-09-22T16:51:22,467 [INFO] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf\_pipeline = pipeline(task=task, model=model\_dir, device=device, \*\*kwargs) 2023-09-22T16:51:22,470 [INFO] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - KeyError: ‘llama’ 2023-09-22T16:51:22,473 [WARN] W-9000-model com.amazonaws.ml.mms.wlm.WorkerLifeCycle - attachIOStreams() threadName=W-model-1 2023-09-22T16:52:29,088 [INFO] pool-2-thread-3 ACCESS\_LOG - /169.254.178.2:42492 “GET /ping HTTP/1.1” 200 0 2023-09-22T16:52:32,706 [INFO] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 2 2023-09-22T16:52:32,706 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Prediction error 2023-09-22T16:52:32,711 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.9/site-packages/transformers/models/auto/configuration\_auto.py”, line 579, in **getitem** 2023-09-22T16:52:32,711 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - KeyError: ‘llama’ 2023-09-22T16:52:32,711 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred: 2023-09-22T16:52:32,712 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.9/site-packages/mms/service.py”, line 108, in predict 2023-09-22T16:52:32,713 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise PredictionException(str(e), 400) 2023-09-22T16:52:32,713 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: ‘llama’ : 400 @philschmid @marshmellow77 Please help. Thanks!! ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [September 25, 2023, 7:46am UTC](https://discuss.huggingface.co/t/modelerror-when-i-run-predict-after-deploying-wizardcoder-for-text-generation/56032/2 "2023-09-25T07:46:08Z")
You should use the LLM container for deploying your model see: [Securely deploy LLMs inside VPCs with Hugging Face and Amazon SageMaker](https://www.philschmid.de/sagemaker-llm-vpc)