# Prepare data for pretraining T5 model

**URL:** <https://discuss.huggingface.co/t/prepare-data-for-pretraining-t5-model/10485>\
**Category:** 🤗Datasets\
**Created:** [October 4, 2021, 7:02am UTC](https://discuss.huggingface.co/t/prepare-data-for-pretraining-t5-model/10485 "2021-10-04T07:02:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![MickyMike](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mickymike/32/15863_2.png) [@MickyMike](https://discuss.huggingface.co/u/MickyMike)\
**Post date:** [October 4, 2021, 7:02am UTC](https://discuss.huggingface.co/t/prepare-data-for-pretraining-t5-model/10485/1 "2021-10-04T07:02:57Z")

</div>

Hi I read the T5 model overview and saw that T5 can be pretrained on LM task using the following code

```auto
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")

input_ids = tokenizer('The <extra_id_0> walks in <extra_id_1> park', return_tensors='pt').input_ids
labels = tokenizer('<extra_id_0> cute dog <extra_id_1> the <extra_id_2>', return_tensors='pt').input_ids
# the forward function automatically creates the correct decoder_input_ids
loss = model(input_ids=input_ids, labels=labels).loss

```

However, I’m confused about how to prepare my text data before doing unsupervised learning on T5  
If I have a .txt, how should I transform my text into input\_ids and labels pair shown in the above code with special tokens as masks (e.g. ‘The \<extra\_id\_0\> walks in \<extra\_id\_1\> park’).

Also, I tried the example T5MLM code provided [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_t5_mlm_flax.py), but there is some issues with my CUDA driver when running code using jax as backend, so the training could only be started on CPU instead of GPU. So I am looking for a way to prepare the data and do pretraining using PyTorch.

Thanks for helping, really appreciate your time  
Michael

---

<div class="post-metadata">

**Author:** ![CRini](https://avatars.discourse-cdn.com/v4/letter/c/f04885/32.png) [@CRini](https://discuss.huggingface.co/u/CRini)\
**Post date:** [May 4, 2023, 4:32am UTC](https://discuss.huggingface.co/t/prepare-data-for-pretraining-t5-model/10485/2 "2023-05-04T04:32:31Z")

</div>

Did you solve your problem? I have the same question