# Trainer API weights initialization

**URL:** <https://discuss.huggingface.co/t/trainer-api-weights-initialization/139886>\
**Category:** 🤗Transformers\
**Created:** [February 9, 2025, 4:55pm UTC](https://discuss.huggingface.co/t/trainer-api-weights-initialization/139886 "2025-02-09T16:55:30Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![fracapuano](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fracapuano/32/27561_2.png) [@fracapuano](https://discuss.huggingface.co/u/fracapuano)\
**Post date:** [February 9, 2025, 4:55pm UTC](https://discuss.huggingface.co/t/trainer-api-weights-initialization/139886/1 "2025-02-09T16:55:30Z")

</div>

Ciao 🤗 I am working (and loving!!!) with the Trainer API to train a custom transformer architecture. So far everything has been fairly smooth, but I was wondering how you guys handle weight initialization.

I can initialize weight as I please by the usual `model.apply(model._init_weights)`, but I was wondering what is the function you use under the hood (if any). Thank you!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 10, 2025, 12:08pm UTC](https://discuss.huggingface.co/t/trainer-api-weights-initialization/139886/2 "2025-02-10T12:08:34Z")

</div>

I think many people initialize with an empty weight or use from\_pretrained to bring in the initial values of the base model.

> [@Lazy model initialization](https://discuss.huggingface.co/t/lazy-model-initialization/20933):
>
> Hello. How can I create a model object and skip the random initialization of weights? The random initialization is time consuming and unnecessary for my case, as I want to load the weights using torch.load\_state\_dict. For instance, see the code below. config = BloomConfig.from\_pretrained("bigscience/bloom") block = BloomBlock(config) # initializes weights randomly, which is time consuming block.load\_state\_dict(torch.load("path\_to\_pytorch\_bin"))

---

<div class="post-metadata">

**Author:** ![fracapuano](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fracapuano/32/27561_2.png) [@fracapuano](https://discuss.huggingface.co/u/fracapuano)\
**Post date:** [February 10, 2025, 1:05pm UTC](https://discuss.huggingface.co/t/trainer-api-weights-initialization/139886/3 "2025-02-10T13:05:44Z")

</div>

Thank you for taking the time to answer @John6666 – Not really sure it applies to my situation tho.

I am interested in weight initialization for training, and specifically on what are the defaults on the Trainer API when it comes down to initialize the weights (I mean, what distributions or is it just a fallback to Pytorch default init)