# Differences between transformers GPT2 and megatron-lm?

**URL:** <https://discuss.huggingface.co/t/differences-between-transformers-gpt2-and-megatron-lm/15554>\
**Category:** 🤗Transformers\
**Created:** [March 10, 2022, 3:01am UTC](https://discuss.huggingface.co/t/differences-between-transformers-gpt2-and-megatron-lm/15554 "2022-03-10T03:01:12Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![lucasjin](https://avatars.discourse-cdn.com/v4/letter/l/f4b2a3/32.png) [@lucasjin](https://discuss.huggingface.co/u/lucasjin)\
**Post date:** [March 10, 2022, 3:01am UTC](https://discuss.huggingface.co/t/differences-between-transformers-gpt2-and-megatron-lm/15554/1 "2022-03-10T03:01:12Z")

</div>

[https://github.com/huggingface/transformers/issues/16016](https://github.com/huggingface/transformers/issues/16016)

Comes from this issue. @valhalla sorry for @ you here again.

My question is:

1. they are all GPT2, but is there any differences? including arch, ops etc.
2. megatron-lm using FusedLayerNorm, but I don’t see such op inside transformers GPT2, is there equal interms of final predictions?
3. what’s the strength of megatron-lm and what’s the weakness compares with transformers?

thank u if anyone could give me a hand.