# Hyperparameter-Tuning on Sagemaker - FP16 parameter not responsive

**URL:** <https://discuss.huggingface.co/t/hyperparameter-tuning-on-sagemaker-fp16-parameter-not-responsive/130113>\
**Category:** Amazon SageMaker\
**Created:** [December 9, 2024, 8:52am UTC](https://discuss.huggingface.co/t/hyperparameter-tuning-on-sagemaker-fp16-parameter-not-responsive/130113 "2024-12-09T08:52:17Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![nicoberk](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nicoberk/32/26983_2.png) [@nicoberk](https://discuss.huggingface.co/u/nicoberk)\
**Post date:** [December 9, 2024, 8:52am UTC](https://discuss.huggingface.co/t/hyperparameter-tuning-on-sagemaker-fp16-parameter-not-responsive/130113/1 "2024-12-09T08:52:17Z")

</div>

Hi there!

I am currently trying to run some hyperparameter tuning on my models using the HyperparameterTuner. However, I fail to vary the FP16 parameter.

I define the search space with

```auto
## define hyperparameter space for tuning
hyperparameter_ranges = {
    "epochs": CategoricalParameter([3, 5, 10]),
    "train_batch_size": CategoricalParameter([16, 32, 64]), # adjust dependent on balanced or not
    "learning_rate": ContinuousParameter(1e-5, 1e-3),
    "warmup_ratio": ContinuousParameter(0, 0.2),
    "weight_decay": ContinuousParameter(0.0, 0.3),
    "fp16": CategoricalParameter([True, False]),
    "seed": IntegerParameter(0, 42),
    "class_weight_factor": ContinuousParameter(0.1, 1.0),
}

```

And pass that to the tuner.

While this mostly runs fine, the fp16 mixed precision does not change over the runs, even though the hyperparameter specification changes:

 ![PastedGraphic-2](https://us1.discourse-cdn.com/hellohellohello/original/3X/5/0/50a5e6cb186eca4bce4a8e1f3c638a69f092ea26.jpeg)

In the training script, I recover the parameters like this:

`parser.add_argument("--fp16", type=bool, default=os.environ["SM_HP_FP16"])`

and pass them to the training arguments like this:

```auto
# define training args
core_args = {...}

extra_args = {}
  
if args.fp16:
    extra_args["fp16"]=args.fp16

training_args = { **core_args,** extra_args}
training_args = TrainingArguments(**training_args)

```

which are then passed to the trainer.

Any idea what the issue might be? I am out of ideas - interestingly it always runs with fp16 even though the default is ‘False’.

Best,  
Nico