# Amazon SageMaker **URL:** https://discuss.huggingface.co/c/sagemaker/17.md [Latest](https://discuss.huggingface.co/latest.md) · [Categories](https://discuss.huggingface.co/categories.md) --- ## [About the Amazon SageMaker category](https://discuss.huggingface.co/t/about-the-amazon-sagemaker-category/4603)
**Author:** [@pierric](https://discuss.huggingface.co/u/pierric)\ **Replies:** 25\ **Last updated:** [August 5, 2021, 3:42pm UTC](https://discuss.huggingface.co/t/about-the-amazon-sagemaker-category/4603 "2021-08-05T15:42:53Z")
This category is for any questions related to using Hugging Face Transformers with Amazon SageMaker. Don’t forget to check the announcement blogpost for more resources. --- ## [HF Sagemaker Setting LLM Parameters](https://discuss.huggingface.co/t/hf-sagemaker-setting-llm-parameters/93234)
**Author:** [@agtm48](https://discuss.huggingface.co/u/agtm48)\ **Replies:** 1\ **Last updated:** [April 13, 2026, 1:31pm UTC](https://discuss.huggingface.co/t/hf-sagemaker-setting-llm-parameters/93234 "2026-04-13T13:31:11Z")
Hi, I’m trying to use Sagemaker’s Batch Transform utility in order to perform LLM inference using a LLAMA-3 8B-Instruct Model. import sagemaker import boto3 from sagemaker.huggingface import HuggingFaceModel, get\_huggin… --- ## [LLM Course - Chapter 1- How to Properly Run Transformers Notebooks in SageMaker Lab?](https://discuss.huggingface.co/t/llm-course-chapter-1-how-to-properly-run-transformers-notebooks-in-sagemaker-lab/170835)
**Author:** [@runtimeterror87](https://discuss.huggingface.co/u/runtimeterror87)\ **Replies:** 2\ **Last updated:** [November 26, 2025, 10:44am UTC](https://discuss.huggingface.co/t/llm-course-chapter-1-how-to-properly-run-transformers-notebooks-in-sagemaker-lab/170835 "2025-11-26T10:44:35Z")
I’m using the SageMaker Lab environment to open and sign in to the Transformers notebook, but I’m not able to run the code directly. I only see an option called “Copy to project”, which sends me to JupyterLab. Do I need… --- ## [Failed pulling HF image on sagemaker as of mid-morning; edit now working](https://discuss.huggingface.co/t/failed-pulling-hf-image-on-sagemaker-as-of-mid-morning-edit-now-working/162398)
**Author:** [@laurb](https://discuss.huggingface.co/u/laurb)\ **Replies:** 0\ **Last updated:** [July 9, 2025, 6:03pm UTC](https://discuss.huggingface.co/t/failed-pulling-hf-image-on-sagemaker-as-of-mid-morning-edit-now-working/162398 "2025-07-09T18:03:49Z")
We have an existing sagemaker endpoint configuration that uses a model we trained with huggingface. It has been working for months and worked earlier this am. However after about 9am edt today we can no longer launch the… --- ## [HuggingFaceModel ignores code directory](https://discuss.huggingface.co/t/huggingfacemodel-ignores-code-directory/159624)
**Author:** [@OM-NS](https://discuss.huggingface.co/u/OM-NS)\ **Replies:** 2\ **Last updated:** [June 17, 2025, 7:47am UTC](https://discuss.huggingface.co/t/huggingfacemodel-ignores-code-directory/159624 "2025-06-17T07:47:11Z")
Hi, I am facing some issues trying to initialize an endpoint with custom inference code for LLaMa 3.2 3B. Specifically, the endpoint launches but it ignores the requirements.txt and inference.py code completely. Here i… --- ## [HuggingFaceModel create fails with no GPU](https://discuss.huggingface.co/t/huggingfacemodel-create-fails-with-no-gpu/159162)
**Author:** [@laurb](https://discuss.huggingface.co/u/laurb)\ **Replies:** 3\ **Last updated:** [June 14, 2025, 12:53am UTC](https://discuss.huggingface.co/t/huggingfacemodel-create-fails-with-no-gpu/159162 "2025-06-14T00:53:30Z")
I have a pretrained custom model with the model.tar.gz file located in S3. I want to create the HuggingFaceModel and the endpoint configuration in Sagemaker. My local machine (where I am running this) does not have a GPU… --- ## [Error deploying BERT on SageMaker](https://discuss.huggingface.co/t/error-deploying-bert-on-sagemaker/8401)
**Author:** [@wsunadawong](https://discuss.huggingface.co/u/wsunadawong)\ **Replies:** 20\ **Last updated:** [April 1, 2025, 4:54pm UTC](https://discuss.huggingface.co/t/error-deploying-bert-on-sagemaker/8401 "2025-04-01T16:54:13Z")
I fine-tuned BERT for text-classification on a custom dataset using HuggingFace and Tensorflow, and now I’m trying to deploy the model for inference through SageMaker. I followed this HuggingFace tutorial but I get the f… --- ## [GGUF BYOC Deployment with AWS SageMaker: \[Errno28\] No space left on device](https://discuss.huggingface.co/t/gguf-byoc-deployment-with-aws-sagemaker-errno28-no-space-left-on-device/147136)
**Author:** [@MatthiasLee](https://discuss.huggingface.co/u/MatthiasLee)\ **Replies:** 1\ **Last updated:** [March 24, 2025, 10:41am UTC](https://discuss.huggingface.co/t/gguf-byoc-deployment-with-aws-sagemaker-errno28-no-space-left-on-device/147136 "2025-03-24T10:41:14Z")
I am trying to deploy deepseek quantized models on AWS SageMaker through this guide that uses the Bring Your Own Container (BYOC) approach: GitHub - aws-samples/deploy-gguf-model-to-sagemaker. Error: \[Errno28\] No space … --- ## [Custom Attributes for Asynchronous Endpoint Sagemaker](https://discuss.huggingface.co/t/custom-attributes-for-asynchronous-endpoint-sagemaker/138638)
**Author:** [@quanamifiable2023](https://discuss.huggingface.co/u/quanamifiable2023)\ **Replies:** 1\ **Last updated:** [February 20, 2025, 11:07am UTC](https://discuss.huggingface.co/t/custom-attributes-for-asynchronous-endpoint-sagemaker/138638 "2025-02-20T11:07:50Z")
I am using AWS Sagemaker to host an asynchronous endpoint. I have a custom inference.py code for this task. This is how i invoke the endpoint using boto3: response = client.invoke\_endpoint\_async( EndpointName='str… --- ## [Sagemaker HuggingFaceModel crashed with CUDA error](https://discuss.huggingface.co/t/sagemaker-huggingfacemodel-crashed-with-cuda-error/40145)
**Author:** [@philhd](https://discuss.huggingface.co/u/philhd)\ **Replies:** 3\ **Last updated:** [February 20, 2025, 11:00am UTC](https://discuss.huggingface.co/t/sagemaker-huggingfacemodel-crashed-with-cuda-error/40145 "2025-02-20T11:00:30Z")
Hi, for inference I created a deployment on AWS which is working fine when used for summarization It works fine on some texts, but as soon as I use a particular long text I get an error and I am not able to run any sum… --- ## [Model \*and\* data parallelism when training on multiple GPUs?](https://discuss.huggingface.co/t/model-and-data-parallelism-when-training-on-multiple-gpus/136835)
**Author:** [@nicoberk](https://discuss.huggingface.co/u/nicoberk)\ **Replies:** 0\ **Last updated:** [January 22, 2025, 9:13am UTC](https://discuss.huggingface.co/t/model-and-data-parallelism-when-training-on-multiple-gpus/136835 "2025-01-22T09:13:18Z")
Hi there! I want to train a model from scratch using a dataset of ~35M online comments. Given the computational costs, I hope to set up the training as efficiently as possible, including parallelization. Following this… --- ## [Cannot import "Conversation" from transformers\_utils.py](https://discuss.huggingface.co/t/cannot-import-conversation-from-transformers-utils-py/91556)
**Author:** [@Vocabua](https://discuss.huggingface.co/u/Vocabua)\ **Replies:** 8\ **Last updated:** [January 9, 2025, 4:14pm UTC](https://discuss.huggingface.co/t/cannot-import-conversation-from-transformers-utils-py/91556 "2025-01-09T16:14:51Z")
I am deploying a Huggingface model with Sagemaker. This is the code that I use to deploy: import sagemaker import boto3 from sagemaker.huggingface import HuggingFaceModel sess = sagemaker.Session() role = sagemaker.get… --- ## [Calling Image Classification Model Deployed in SageMaker Endpoint](https://discuss.huggingface.co/t/calling-image-classification-model-deployed-in-sagemaker-endpoint/32200)
**Author:** [@TBRE](https://discuss.huggingface.co/u/TBRE)\ **Replies:** 20\ **Last updated:** [January 3, 2025, 4:04pm UTC](https://discuss.huggingface.co/t/calling-image-classification-model-deployed-in-sagemaker-endpoint/32200 "2025-01-03T16:04:06Z")
Hi Community! I am deploying this HF model (EdBianchi/vit-fire-detection · Hugging Face) on AWS SageMaker. Once deployed, I have an S3 bucket where I am uploading .jpg images too. This S3 bucket triggers a Lambda funct… --- ## [HuggingFaceModel() Sagemaker invoke error in Stable Diffusion 2.1](https://discuss.huggingface.co/t/huggingfacemodel-sagemaker-invoke-error-in-stable-diffusion-2-1/131957)
**Author:** [@Sanan](https://discuss.huggingface.co/u/Sanan)\ **Replies:** 0\ **Last updated:** [December 17, 2024, 8:27pm UTC](https://discuss.huggingface.co/t/huggingfacemodel-sagemaker-invoke-error-in-stable-diffusion-2-1/131957 "2024-12-17T20:27:17Z")
Hello, I deployed stabilityai/stable-diffusion-2-1 model on Sagemaker for text to image task however when I use invoke for getting output I receive following error: File “/app/main.py”, line 265, in invoke\_model resp… --- ## [Hyperparameter-Tuning on Sagemaker - FP16 parameter not responsive](https://discuss.huggingface.co/t/hyperparameter-tuning-on-sagemaker-fp16-parameter-not-responsive/130113)
**Author:** [@nicoberk](https://discuss.huggingface.co/u/nicoberk)\ **Replies:** 0\ **Last updated:** [December 9, 2024, 8:52am UTC](https://discuss.huggingface.co/t/hyperparameter-tuning-on-sagemaker-fp16-parameter-not-responsive/130113 "2024-12-09T08:52:17Z")
Hi there! I am currently trying to run some hyperparameter tuning on my models using the HyperparameterTuner. However, I fail to vary the FP16 parameter. I define the search space with ## define hyperparameter space f… --- ## [Clean up after calling Estimator](https://discuss.huggingface.co/t/clean-up-after-calling-estimator/128847)
**Author:** [@zilong](https://discuss.huggingface.co/u/zilong)\ **Replies:** 0\ **Last updated:** [December 2, 2024, 1:29pm UTC](https://discuss.huggingface.co/t/clean-up-after-calling-estimator/128847 "2024-12-02T13:29:17Z")
I am using the huggingface estimator to train my model and use managed spot instance. I notice that the checkpoints persist after starting the whole training process from scratch. How do I manage this checkpoint data fo… --- ## [Deploying Tencent Hunyuan model](https://discuss.huggingface.co/t/deploying-tencent-hunyuan-model/116193)
**Author:** [@Alex1712](https://discuss.huggingface.co/u/Alex1712)\ **Replies:** 1\ **Last updated:** [November 27, 2024, 3:18pm UTC](https://discuss.huggingface.co/t/deploying-tencent-hunyuan-model/116193 "2024-11-27T15:18:30Z")
Hi! I wanted to deploy the new Tencent Hunyuan MoE model using sagemaker to test it. This is my code: import json import sagemaker import boto3 from sagemaker.huggingface import HuggingFaceModel, get\_huggingface\_llm\_im… --- ## [Multi-lora serving with adapters on S3](https://discuss.huggingface.co/t/multi-lora-serving-with-adapters-on-s3/117629)
**Author:** [@lpvitrai](https://discuss.huggingface.co/u/lpvitrai)\ **Replies:** 0\ **Last updated:** [November 12, 2024, 8:55pm UTC](https://discuss.huggingface.co/t/multi-lora-serving-with-adapters-on-s3/117629 "2024-11-12T20:55:42Z")
I’ve read this great article and want to deploy a base model (Qwen2.5 7B) and use it with different adapters. My model is fine-tuned via SageMaker and saved in S3. I created the model like this: config = { 'HF\_MODEL\_… --- ## [Streaming output text When deploying a finetuned (SFT, DPO) model with custom inference script](https://discuss.huggingface.co/t/streaming-output-text-when-deploying-a-finetuned-sft-dpo-model-with-custom-inference-script/116214)
**Author:** [@xnaxi](https://discuss.huggingface.co/u/xnaxi)\ **Replies:** 1\ **Last updated:** [November 8, 2024, 7:10am UTC](https://discuss.huggingface.co/t/streaming-output-text-when-deploying-a-finetuned-sft-dpo-model-with-custom-inference-script/116214 "2024-11-08T07:10:04Z")
I have a finetuned mistral instruct V2 model that has been fine-tuned using SFT followed by DPO mechanisms. I want to deploy this on sagemaker with custom inference script. While doing this I also want to stream the outp… --- ## [Async TEI Deployment Cannot Handle Requests greater than 2mb](https://discuss.huggingface.co/t/async-tei-deployment-cannot-handle-requests-greater-than-2mb/107529)
**Author:** [@mavjan](https://discuss.huggingface.co/u/mavjan)\ **Replies:** 2\ **Last updated:** [November 4, 2024, 7:47am UTC](https://discuss.huggingface.co/t/async-tei-deployment-cannot-handle-requests-greater-than-2mb/107529 "2024-11-04T07:47:25Z")
We are sending in this config: 'HF\_MODEL\_ID':'BAAI/bge-m3', "DTYPE": "float16", "MAX\_BATCH\_TOKENS": "163840000", "MAX\_CLIENT\_BATCH\_SIZE": "320000", "PAYLOAD\_LIMIT": "200000000", "MAX\_BATCH\_REQUES… --- ## [Unable to deploy Qwen2-VL model on SageMaker](https://discuss.huggingface.co/t/unable-to-deploy-qwen2-vl-model-on-sagemaker/113624)
**Author:** [@hubkrieb-eifer](https://discuss.huggingface.co/u/hubkrieb-eifer)\ **Replies:** 3\ **Last updated:** [October 24, 2024, 8:54am UTC](https://discuss.huggingface.co/t/unable-to-deploy-qwen2-vl-model-on-sagemaker/113624 "2024-10-24T08:54:41Z")
I want to deploy Qwen2-VL-2B-Instruct to a SageMaker endpoint. I know that the model needs additional requirements such as qwen-vl-utils but also needs the latest version of transformers (4.45.2) to work properly. As thi… --- ## [Vicuan error on Sagemaker](https://discuss.huggingface.co/t/vicuan-error-on-sagemaker/42301)
**Author:** [@ozcode](https://discuss.huggingface.co/u/ozcode)\ **Replies:** 3\ **Last updated:** [October 23, 2024, 11:21am UTC](https://discuss.huggingface.co/t/vicuan-error-on-sagemaker/42301 "2024-10-23T11:21:01Z")
Hi folks, I have been trying to deploy TheBloke/vicuna-7B-1.1-HF to SageMaker but with no luck. I have not had problems with other models like bloom-3b. I used the following code to deploy: import sagemaker import boto… --- ## [VRAM Usage Differences in SageMaker Training Jobs vs. Direct Instance for Fine-Tuning LLama3 8B with QLoRA](https://discuss.huggingface.co/t/vram-usage-differences-in-sagemaker-training-jobs-vs-direct-instance-for-fine-tuning-llama3-8b-with-qlora/112624)
**Author:** [@ms3141592](https://discuss.huggingface.co/u/ms3141592)\ **Replies:** 0\ **Last updated:** [October 18, 2024, 6:49am UTC](https://discuss.huggingface.co/t/vram-usage-differences-in-sagemaker-training-jobs-vs-direct-instance-for-fine-tuning-llama3-8b-with-qlora/112624 "2024-10-18T06:49:52Z")
Hi, everyone. I’m currently attempting to fine-tune meta-llama/Meta-Llama-3-8B-Instruct model using QLoRA. I successfully completed the fine-tuning using an Amazon SageMaker Training Job on a ml.g5.8xlarge instance. H… --- ## [Model\_fn and predict\_fn called multiple times?](https://discuss.huggingface.co/t/model-fn-and-predict-fn-called-multiple-times/32117)
**Author:** [@Abesadi](https://discuss.huggingface.co/u/Abesadi)\ **Replies:** 2\ **Last updated:** [October 11, 2024, 8:16am UTC](https://discuss.huggingface.co/t/model-fn-and-predict-fn-called-multiple-times/32117 "2024-10-11T08:16:17Z")
I am using the prebuilt SageMaker Hugging face container In the entry\_point,I include a script which carries out the batch transformation job. ‘’’ def model\_fn(): … def input\_fn(): … def predict\_fn(): ‘’’ --- ## [Need help deploying a HF model to AWS Sagemaker](https://discuss.huggingface.co/t/need-help-deploying-a-hf-model-to-aws-sagemaker/108794)
**Author:** [@bhashwarsengupta](https://discuss.huggingface.co/u/bhashwarsengupta)\ **Replies:** 2\ **Last updated:** [September 27, 2024, 9:20am UTC](https://discuss.huggingface.co/t/need-help-deploying-a-hf-model-to-aws-sagemaker/108794 "2024-09-27T09:20:27Z")
I’m trying to deploy a private HF model to AWS Sagemaker by executing the following code locally: import sagemaker import boto3 from sagemaker.huggingface import HuggingFaceModel from huggingface\_hub import login from d… --- ## [LLM with 1048k hosted on sagemaker](https://discuss.huggingface.co/t/llm-with-1048k-hosted-on-sagemaker/106309)
**Author:** [@Janekowal512](https://discuss.huggingface.co/u/Janekowal512)\ **Replies:** 0\ **Last updated:** [September 11, 2024, 10:19am UTC](https://discuss.huggingface.co/t/llm-with-1048k-hosted-on-sagemaker/106309 "2024-09-11T10:19:11Z")
I downloaded this Llama3 70B model (gradientai/Llama-3-70B-Instruct-Gradient-1048k · Hugging Face) which can handle 1048k context length. I want to deploy this model to sagemaker real-time endpoint with ml.g5.48xlarge i… --- ## [Need help deploying a model on AWS SageMaker](https://discuss.huggingface.co/t/need-help-deploying-a-model-on-aws-sagemaker/105917)
**Author:** [@paco987654](https://discuss.huggingface.co/u/paco987654)\ **Replies:** 0\ **Last updated:** [September 9, 2024, 1:48am UTC](https://discuss.huggingface.co/t/need-help-deploying-a-model-on-aws-sagemaker/105917 "2024-09-09T01:48:01Z")
Hello guys, I’m trying to deploy a Llama model on SageMaker but uh… it’s proving to be more technically advanced than I can handle. So I wanted to ask if anyone would be willing to guide me through it or deploy it for … --- ## [AWS SageMaker Endpoint Error | Mistral 7B Instruct v3 :](https://discuss.huggingface.co/t/aws-sagemaker-endpoint-error-mistral-7b-instruct-v3/103622)
**Author:** [@NormalMatt](https://discuss.huggingface.co/u/NormalMatt)\ **Replies:** 0\ **Last updated:** [August 23, 2024, 12:23am UTC](https://discuss.huggingface.co/t/aws-sagemaker-endpoint-error-mistral-7b-instruct-v3/103622 "2024-08-23T00:23:50Z")
Finetuned Mistral 7B Instruct v3 having issues deployed to AWS Endpoint using Sagemaker HuggingFace library. Exception: data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 6952 column 3 Er… --- ## [Pytorch 2.2.0 release of AWS deep learning containers](https://discuss.huggingface.co/t/pytorch-2-2-0-release-of-aws-deep-learning-containers/102969)
**Author:** [@ojturner](https://discuss.huggingface.co/u/ojturner)\ **Replies:** 0\ **Last updated:** [August 19, 2024, 11:15am UTC](https://discuss.huggingface.co/t/pytorch-2-2-0-release-of-aws-deep-learning-containers/102969 "2024-08-19T11:15:46Z")
Hi there, We make use of the HuggingFace training and inference containers for use with AWS SageMaker. Specifically those listed on the available\_images.md page. Recently 2 high-severity CVEs (CVE-2024-31580 and CVE-20… --- ## [Using pipeline parameters with Sagemaker](https://discuss.huggingface.co/t/using-pipeline-parameters-with-sagemaker/102351)
**Author:** [@sbarnettGE](https://discuss.huggingface.co/u/sbarnettGE)\ **Replies:** 0\ **Last updated:** [August 14, 2024, 1:12pm UTC](https://discuss.huggingface.co/t/using-pipeline-parameters-with-sagemaker/102351 "2024-08-14T13:12:19Z")
I’m struggling to pass pipeline parameters to a model hosted on sagemaker. I’m trying to deploy a token-classification model which is stored in S3 as a model.tar.gz with the ‘stride’ parameter so that it does the automat… [Next page](https://discuss.huggingface.co/c/sagemaker/17.md?page=1)