# Pretokenization of dataset for finetuning

**URL:** <https://discuss.huggingface.co/t/pretokenization-of-dataset-for-finetuning/157419>\
**Category:** 🤗Datasets\
**Created:** [May 30, 2025, 7:26pm UTC](https://discuss.huggingface.co/t/pretokenization-of-dataset-for-finetuning/157419 "2025-05-30T19:26:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![starsofchance](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/starsofchance/32/44937_2.png) [@starsofchance](https://discuss.huggingface.co/u/starsofchance)\
**Post date:** [May 30, 2025, 7:26pm UTC](https://discuss.huggingface.co/t/pretokenization-of-dataset-for-finetuning/157419/1 "2025-05-30T19:26:55Z")

</div>

Hello friends,

I have a dataset (Prime\_Vul) for fine-tuning.  
The data format is as follows: desired input and output:  
JSON

{  
“idx”: 201849,  
“input”: “void sqlite3Pragma( Parse \*pParse, Token \*pId1, …rest of the code for input }”,  
“output”: {  
“is\_vulnerable”: true,  
“vulnerability\_types”: [“Improper Check for Unusual or Exceptional Conditions”],  
“explanation”: “pragma.c in SQLite through 3.30.1 mishandles NOT NULL in an integrity\_check PRAGMA command in certain cases of generated columns.”,  
“severity\_level”: “NoInfo”,  
“cwe”: [“CWE-754”],  
“cve”: “CVE-2019-19646”  
},  
“code\_token\_length”: 19000,  
“total\_token\_length”: 20042,  
“max\_tokens\_setting”: 32768  
}

My question is, if I want to tokenize this dataset and then use it for the fine-tuning process, which parts exactly should I tokenize?  
All the data? Or just the input?  
Please guide me.

---

<div class="post-metadata">

**Author:** ![Mdrnfox](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mdrnfox/32/47695_2.png) [@Mdrnfox](https://discuss.huggingface.co/u/Mdrnfox)\
**Post date:** [May 30, 2025, 8:04pm UTC](https://discuss.huggingface.co/t/pretokenization-of-dataset-for-finetuning/157419/2 "2025-05-30T20:04:47Z")

</div>

> [@starsofchance](#):
>
> All the data? Or just the input?

you need to tokenize everything that the model will see and learn to predict. The output text must also be tokenized so it can appear in the model’s decoder during training and contribute to the loss.

Hope that helps 🙂

---

<div class="post-metadata">

**Author:** ![starsofchance](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/starsofchance/32/44937_2.png) [@starsofchance](https://discuss.huggingface.co/u/starsofchance)\
**Post date:** [May 30, 2025, 8:58pm UTC](https://discuss.huggingface.co/t/pretokenization-of-dataset-for-finetuning/157419/3 "2025-05-30T20:58:36Z")

</div>

thank you so much.  
yes that helped a lot.  
but now i have more question, i would appreciate if you help me with these too.  
1: should i also tokenise the labels?  
2: i should use padding correct?  
so in every batch of dataset (different maximum token length) the input sequence be the same length for every function the code receives during fine tuning?  
3: should i also use padding on the output?  
4: and if yes, should i padd the input and output separately? or as one?  
Again, thank you for the first reply.

---

<div class="post-metadata">

**Author:** ![Mdrnfox](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mdrnfox/32/47695_2.png) [@Mdrnfox](https://discuss.huggingface.co/u/Mdrnfox)\
**Post date:** [May 30, 2025, 9:05pm UTC](https://discuss.huggingface.co/t/pretokenization-of-dataset-for-finetuning/157419/4 "2025-05-30T21:05:17Z")

</div>

> [@starsofchance](#):
>
> 1: should i also tokenise the labels?  
> 2: i should use padding correct?  
> so in every batch of dataset (different maximum token length) the input sequence be the same length for every function the code receives during fine tuning?  
> 3: should i also use padding on the output?  
> 4: and if yes, should i padd the input and output separately? or as one?

I would tokenize the necessary labels, you can drop unneccessary labels. i would tokenize the prompt and completion is. Then you can carve out where the tokens get delegated. I would use a max length padding. Yes, the padding must match the padding shape of your input padding.

Something like this batch = tokenizer(  
prompts,  
completions,  
padding=“longest”,  
truncation=True,  
return\_tensors=“pt”  
)

---

<div class="post-metadata">

**Author:** ![starsofchance](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/starsofchance/32/44937_2.png) [@starsofchance](https://discuss.huggingface.co/u/starsofchance)\
**Post date:** [May 31, 2025, 10:10am UTC](https://discuss.huggingface.co/t/pretokenization-of-dataset-for-finetuning/157419/5 "2025-05-31T10:10:44Z")

</div>

Thank you so so much for the answer.  
It was a huge help to me.