# Linear Learning Rate Warmup with step-decay

**URL:** <https://discuss.huggingface.co/t/linear-learning-rate-warmup-with-step-decay/5604>\
**Category:** Beginners\
**Created:** [April 19, 2021, 5:00am UTC](https://discuss.huggingface.co/t/linear-learning-rate-warmup-with-step-decay/5604 "2021-04-19T05:00:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![adaptivedecay](https://avatars.discourse-cdn.com/v4/letter/a/4af34b/32.png) [@adaptivedecay](https://discuss.huggingface.co/u/adaptivedecay)\
**Post date:** [April 19, 2021, 5:00am UTC](https://discuss.huggingface.co/t/linear-learning-rate-warmup-with-step-decay/5604/1 "2021-04-19T05:00:00Z")

</div>

Hey Guys,

I am new to PyTorch and have to train deep models like ResNet-_, VGG-_ which requires the following learning rate schedule:

1. Linearly increase the learning rate from 0 to ‘initial\_lr’ in the first k training steps/iterations

2. Continue with ‘initial\_lr’ for the next ‘m’ training steps

3. Decay the learning rate in a step-decay manner. For example, say after 30th epoch, you reduce the ‘initial\_lr’ by 10. And after 45th epoch, you further reduce it by 10 for any further training.

This can be better visualized using the following picture:  
 ![lr](https://us1.discourse-cdn.com/hellohellohello/original/2X/2/2ae89530c19b3928543cfdbd0611f8eac0e8f385.png)

This is an example using LeNet-300-100 on MNIST with TensorFlow2.

How can I achieve this particular learning rate schedule with ‘huggingface’?

Thanks

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [April 19, 2021, 7:42am UTC](https://discuss.huggingface.co/t/linear-learning-rate-warmup-with-step-decay/5604/2 "2021-04-19T07:42:20Z")

</div>

hey @adaptivedecay, you can define your own scheduler for the learning rate by subclassing `Trainer` and overriding the `create_scheduler` function to include your logic: [Trainer — transformers 4.5.0.dev0 documentation](https://huggingface.co/transformers/main_classes/trainer.html?highlight=trainer#transformers.Trainer.create_scheduler)

alternatively, you can pass the optimizer and scheduler as a tuple in the `optimizers` argument.

---

<div class="post-metadata">

**Author:** ![adaptivedecay](https://avatars.discourse-cdn.com/v4/letter/a/4af34b/32.png) [@adaptivedecay](https://discuss.huggingface.co/u/adaptivedecay)\
**Post date:** [April 19, 2021, 7:59am UTC](https://discuss.huggingface.co/t/linear-learning-rate-warmup-with-step-decay/5604/3 "2021-04-19T07:59:01Z")

</div>

Hi @lewtun, any sample code would be helpful?

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [April 19, 2021, 8:21am UTC](https://discuss.huggingface.co/t/linear-learning-rate-warmup-with-step-decay/5604/4 "2021-04-19T08:21:08Z")

</div>

you can take a look at the implementation in the `Trainer` here: [transformers/trainer.py at d9c62047a8d75e18d2849d345ab3394875a712ef · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/d9c62047a8d75e18d2849d345ab3394875a712ef/src/transformers/trainer.py#L766)

---

<div class="post-metadata">

**Author:** ![adaptivedecay](https://avatars.discourse-cdn.com/v4/letter/a/4af34b/32.png) [@adaptivedecay](https://discuss.huggingface.co/u/adaptivedecay)\
**Post date:** [April 21, 2021, 1:22pm UTC](https://discuss.huggingface.co/t/linear-learning-rate-warmup-with-step-decay/5604/5 "2021-04-21T13:22:43Z")

</div>

Thanks for your link. I will have a look into it and get back!