# HF Sagemaker Setting LLM Parameters

**URL:** <https://discuss.huggingface.co/t/hf-sagemaker-setting-llm-parameters/93234>\
**Category:** Amazon SageMaker\
**Created:** [June 20, 2024, 5:51pm UTC](https://discuss.huggingface.co/t/hf-sagemaker-setting-llm-parameters/93234 "2024-06-20T17:51:47Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![agtm48](https://avatars.discourse-cdn.com/v4/letter/a/9fc29f/32.png) [@agtm48](https://discuss.huggingface.co/u/agtm48)\
**Post date:** [June 20, 2024, 5:51pm UTC](https://discuss.huggingface.co/t/hf-sagemaker-setting-llm-parameters/93234/1 "2024-06-20T17:51:47Z")

</div>

Hi, I’m trying to use Sagemaker’s Batch Transform utility in order to perform LLM inference using a LLAMA-3 8B-Instruct Model.

```auto
import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri
import json
generate_parameters = {
    'temperature': '0.6',
    'top_p': '0.9',
    'do_sample': 'True',
    'max_new_tokens': '256',
    'return_full_text': 'False' # This ensures the input text is not included in the output
}
# hub.update({'HF_PARAMETERS': json.dumps(generate_parameters)})

huggingface_model = HuggingFaceModel(
    env=hub, # configuration for loading model from Hub
    role=role,
    image_uri=get_huggingface_llm_image_uri("huggingface", version="2.0.2"),
)

batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g5.2xlarge',
    output_path=s3_output_data_path,
    strategy='SingleRecord',
    env = generate_parameters # Max payload size in MB
    )

batch_job.transform(
    data=s3_input_data_path,
    content_type='application/json',
    split_type='Line'
)

```

No matter what I do, I just can’t get this to properly set the LLM parameters. The temperature, top\_p, etc. are all just defaulted to None.

I’d appreciate it if someone could maybe take a look at this and see if I’m passing the params in wrong?

---

<div class="post-metadata">

**Author:** ![JerroldK](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jerroldk/32/56341_2.png) [@JerroldK](https://discuss.huggingface.co/u/JerroldK)\
**Post date:** [April 13, 2026, 1:31pm UTC](https://discuss.huggingface.co/t/hf-sagemaker-setting-llm-parameters/93234/2 "2026-04-13T13:31:11Z")

</div>

After a lot of scouring on google for this exact problem I found an example that may be the solution you are looking for on a sagemaker-examples.

You need to nest the parameters into another dictionary

```auto
hyper_params = {"max_length": 30, "top_k": 50, "top_p": 0.95, "do_sample": True}
hyper_params_dict = {"HYPER_PARAMS": str(hyper_params)}

```

> **[max\_length - SageMaker JumpStart Foundation Models - HuggingFace Text2Text...](https://sagemaker-examples.readthedocs.io/en/latest/introduction_to_amazon_algorithms/jumpstart-foundation-models/text2text-generation-Batch-Transform.html#4.-Specify-Batch-Transform-Job-HyperParameters)**
>
> This notebook’s CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.