# T5 training from scratch

**URL:** <https://discuss.huggingface.co/t/t5-training-from-scratch/1898>\
**Category:** Beginners\
**Created:** [November 4, 2020, 5:42pm UTC](https://discuss.huggingface.co/t/t5-training-from-scratch/1898 "2020-11-04T17:42:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon47807654](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@anon47807654](https://discuss.huggingface.co/u/anon47807654)\
**Post date:** [November 4, 2020, 5:42pm UTC](https://discuss.huggingface.co/t/t5-training-from-scratch/1898/1 "2020-11-04T17:42:33Z")

</div>

Hi all,  
I would like to train a T5 model (t5-base version) without loading the pretrained weights, if I write the following:  
from transformers import T5Config, T5Model  
config = T5Config.from\_pretrained(‘t5-base’)  
model = T5Model(config)  
It will produce the t5-base sized T5 model without loading the checkpoint weights?  
Thanks

---

<div class="post-metadata">

**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](https://discuss.huggingface.co/u/Jung)\
**Post date:** [November 4, 2020, 11:16pm UTC](https://discuss.huggingface.co/t/t5-training-from-scratch/1898/2 "2020-11-04T23:16:10Z")

</div>

> [@anon47807654](#):
>
> from transformers import T5Config, T5Model  
> config = T5Config.from\_pretrained(‘t5-base’)  
> model = T5Model(config)

Yes, that’s correct. You will definitely see the “pretrained weights” download progress bar in the other case 🙂

---

<div class="post-metadata">

**Author:** ![anon47807654](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@anon47807654](https://discuss.huggingface.co/u/anon47807654)\
**Post date:** [November 5, 2020, 9:26am UTC](https://discuss.huggingface.co/t/t5-training-from-scratch/1898/3 "2020-11-05T09:26:35Z")

</div>

Thanks for your reply, if I will see the download progress bar it means that the pretrained weights have been loaded?  
The fact that the T5Config was initialized by “from\_pretrained” doesn’t imply downloading something?

---

<div class="post-metadata">

**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](https://discuss.huggingface.co/u/Jung)\
**Post date:** [November 5, 2020, 9:50am UTC](https://discuss.huggingface.co/t/t5-training-from-scratch/1898/4 "2020-11-05T09:50:23Z")

</div>

That’s only config (eg. hyperparameters) 😉

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [November 5, 2020, 7:01pm UTC](https://discuss.huggingface.co/t/t5-training-from-scratch/1898/5 "2020-11-05T19:01:02Z")

</div>

> [@anon47807654](#):
>
> been loaded?  
> The fact that the T5Config was initialized by “from\_pretrained” doesn’t imply downloading something?

There will be a little progress bar of \< 1mb. The weights are \> 1GB.

---

<div class="post-metadata">

**Author:** ![anon47807654](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@anon47807654](https://discuss.huggingface.co/u/anon47807654)\
**Post date:** [November 5, 2020, 7:44pm UTC](https://discuss.huggingface.co/t/t5-training-from-scratch/1898/6 "2020-11-05T19:44:05Z")

</div>

Yeah, I’ve noticed it. Thank you all!