# Training speed vs Megatron

**URL:** <https://discuss.huggingface.co/t/training-speed-vs-megatron/63989>\
**Category:** 🤗Transformers\
**Created:** [November 29, 2023, 11:17am UTC](https://discuss.huggingface.co/t/training-speed-vs-megatron/63989 "2023-11-29T11:17:50Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![xianf](https://avatars.discourse-cdn.com/v4/letter/x/a3d4f5/32.png) [@xianf](https://discuss.huggingface.co/u/xianf)\
**Post date:** [November 29, 2023, 11:17am UTC](https://discuss.huggingface.co/t/training-speed-vs-megatron/63989/1 "2023-11-29T11:17:50Z")

</div>

I try to pretrain a language model so I made some speed tests about transformers and Megatron. I found if I pre-tokenize the data and use flash-attention, the training speed is on par with Megatron in one node with 8 GPU.  
Is this correct? As I know, transformers is slower than Megatron, especially for pretrain from scratch.