# InvokeEndpoint Error : Predict function Invocation Timeout **URL:** \ **Category:** šŸ¤—Transformers\ **Created:** [March 27, 2023, 6:06am UTC](https://discuss.huggingface.co/t/invokeendpoint-error-predict-function-invocation-timeout/34755 "2023-03-27T06:06:00Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Arsh25](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arsh25/32/14961_2.png) [@Arsh25](https://discuss.huggingface.co/u/Arsh25)\ **Post date:** [March 27, 2023, 6:06am UTC](https://discuss.huggingface.co/t/invokeendpoint-error-predict-function-invocation-timeout/34755/1 "2023-03-27T06:06:00Z")
I am trying to use AWS S3 option to load the hugging face transformer model GPT-NeoXT-Chat-Base-20B. The endpoint at SageMaker is successfully created. predictor = huggingface\_model.deploy( initial\_instance\_count=1, instance\_type=ā€œml.g5.12xlargeā€, ModelDataDownloadTimeoutInSeconds = 2400, ContainerStartupHealthCheckTimeoutInSeconds = 2400, endpoint\_name=endpoint\_name ) While calling the endpoint, getting the invocation timeout. By default, I guess its 1 min, how to increase the timeout interval as the prediction might take more than 1 min? Error while calling predict function: Code: predictor.predict({ā€˜inputsā€™: "Can you please let us know more details about your "}) Error : File ~/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py:960, in BaseClient.\_make\_api\_call(self, operation\_name, api\_params) 958 error\_code = parsed\_response.get(ā€œErrorā€, {}).get(ā€œCodeā€) 959 error\_class = self.exceptions.from\_code(error\_code) ā†’ 960 raise error\_class(parsed\_response, operation\_name) 961 else: 962 return parsed\_response ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (0) from primary with message ā€œYour invocation timed out while waiting for a response from container primary. Review the latency metrics for each container in Amazon CloudWatch, resolve the issue, and try again.ā€. See [https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logEventViewer:group=/aws/sagemaker/Endpoints/GPT-Neo-XT-Test-2023-03-27-05-32-22-539](https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logEventViewer:group=/aws/sagemaker/Endpoints/GPT-Neo-XT-Test-2023-03-27-05-32-22-539) in account 597748488783 for more information. Please suggest any pointers to proceed. ---
**Author:** ![carterrees](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/carterrees/32/14179_2.png) [@carterrees](https://discuss.huggingface.co/u/carterrees)\ **Post date:** [April 7, 2023, 2:28am UTC](https://discuss.huggingface.co/t/invokeendpoint-error-predict-function-invocation-timeout/34755/2 "2023-04-07T02:28:58Z")
Having the same issue with all of the GPT models on AWS. BERT based models work just fine. ---
**Author:** ![carterrees](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/carterrees/32/14179_2.png) [@carterrees](https://discuss.huggingface.co/u/carterrees)\ **Post date:** [April 7, 2023, 2:38am UTC](https://discuss.huggingface.co/t/invokeendpoint-error-predict-function-invocation-timeout/34755/3 "2023-04-07T02:38:18Z")
Update. I was able to get this to run properly. I wonder if it has to do with the instance size assigned to the predictor. I was taking the default ml.m5.xlarge at first and then tried larger machines. The size you see below finally allowed me to invoke the endpoint. `# deploy model to SageMaker Inference predictor = huggingface_model.deploy( initial_instance_count=1, # number of instances instance_type='ml.p3.2xlarge' # ec2 instance type )` ---
**Author:** ![caitriggs](https://avatars.discourse-cdn.com/v4/letter/c/f1d935/32.png) [@caitriggs](https://discuss.huggingface.co/u/caitriggs)\ **Post date:** [December 1, 2023, 7:32pm UTC](https://discuss.huggingface.co/t/invokeendpoint-error-predict-function-invocation-timeout/34755/4 "2023-12-01T19:32:51Z")
Iā€™m using a `ml.g5.12xlarge` for my LLM which is the same size that runs other foundation LLM models of the same param size (ie JumpStart verisions) and Iā€™m still getting the timeout error. So I donā€™t think itā€™s just the size of the instance (Using a fine-tuned falcon-7b model deployed to a ml.g5.12xlarge which is the same size compute i was able to fine-tune the model on) Iā€™m guessing it might have to do with how inference is called. For instance, in [this SageMaker Pipelines example](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-pipelines/tabular/train-register-deploy-pipeline-model/train%20register%20and%20deploy%20a%20pipeline%20model.ipynb) during ā€œDefine a Register Model Stepā€¦ā€ we just pass an image to use for inference and set image\_scope=ā€œinferenceā€ and the Model object the model data (but no inference script itself). Later during ā€œDeploy latest approved model to a real-time endpointā€ we grab the Approved model\_package\_arn and deploy the model to an endpoint with `model.deploy(initial_instance_count=x, instance_type="compute type", endpoint_name="endpoint name")` but I still never see where we tell the endpoint or registered Model how to inference. Is there a black box evaluate/predict method that sagemaker is defaulted to which just doesnā€™t work yet for HuggingFace LLM type models? Investigatingā€¦