# Pre-training googlebyt5small

**URL:** <https://discuss.huggingface.co/t/pre-training-googlebyt5small/25028>\
**Category:** 🤗Transformers\
**Created:** [October 26, 2022, 3:52pm UTC](https://discuss.huggingface.co/t/pre-training-googlebyt5small/25028 "2022-10-26T15:52:57Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![antoine2323231](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@antoine2323231](https://discuss.huggingface.co/u/antoine2323231)\
**Post date:** [October 26, 2022, 3:52pm UTC](https://discuss.huggingface.co/t/pre-training-googlebyt5small/25028/1 "2022-10-26T15:52:57Z")

</div>

Is it possible to use the run\_t5\_mlm\_flax.py script (link below) in order to pre-train the googlebyt5small and google/t5-small-lm-adapt models. Or there is a different script in order to do so?

> **[transformers/examples/flax/language-modeling at main · huggingface/transformers](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling)**
>
> main/examples/flax/language-modeling

Thanks in advance…