# Setting optimizer parameters with DeepSpeed

**URL:** <https://discuss.huggingface.co/t/setting-optimizer-parameters-with-deepspeed/70143>\
**Category:** 🤗Accelerate\
**Created:** [January 22, 2024, 8:29pm UTC](https://discuss.huggingface.co/t/setting-optimizer-parameters-with-deepspeed/70143 "2024-01-22T20:29:26Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![normster](https://avatars.discourse-cdn.com/v4/letter/n/2bfe46/32.png) [@normster](https://discuss.huggingface.co/u/normster)\
**Post date:** [January 22, 2024, 8:29pm UTC](https://discuss.huggingface.co/t/setting-optimizer-parameters-with-deepspeed/70143/1 "2024-01-22T20:29:26Z")

</div>

I’ve been using DeepSpeed with accelerate to launch the Transformers standard trainer without specifying a json config file for DeepSpeed. I just noticed this snipped in the logs:

```auto
[INFO|deepspeed.py:303] 2024-01-22 12:27:21,324 >> Detected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)
Using /home/norman/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...
Using /home/norman/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...
Using /home/norman/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/norman/.cache/torch_extensions/py311_cu121/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Using /home/norman/.cache/torch_extensions/py311_cu121 as PyTorch extensions root...
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.933143138885498 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.970329523086548 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.9743385314941406 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.976152181625366 seconds
Adam Optimizer #0 is created with AVX512 arithmetic capability.
Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.010000, adam_w=1

```

Even though I’ve set weight decay to 0.0 through the command line (i.e. `accelerate launch ... --weight_decay 0.1`).

Are the logs accurate? Is the weight decay value used in the optimizer 0.01 or 0.0? Do I need to set the weight decay through an explicit json config for DeepSpeed?