# Tokenize a large corpus

**URL:** <https://discuss.huggingface.co/t/tokenize-a-large-corpus/99348>\
**Category:** Intermediate\
**Created:** [July 25, 2024, 9:55am UTC](https://discuss.huggingface.co/t/tokenize-a-large-corpus/99348 "2024-07-25T09:55:00Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![fabiancpl](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fabiancpl/32/29707_2.png) [@fabiancpl](https://discuss.huggingface.co/u/fabiancpl)\
**Post date:** [July 25, 2024, 9:55am UTC](https://discuss.huggingface.co/t/tokenize-a-large-corpus/99348/1 "2024-07-25T09:55:00Z")

</div>

Hi guys,

How can I efficiently tokenize a corpus of 340M of sentences for pre-training models like BERT or GPT-2? Following the standar documentation, the process will take around 20 hours to finish.

Thanks.