# How to run trainer.py with megatron\_lm\_plugin **URL:** \ **Category:** 🤗Transformers\ **Created:** [August 15, 2023, 8:27am UTC](https://discuss.huggingface.co/t/how-to-run-trainer-py-with-megatron-lm-plugin/50804 "2023-08-15T08:27:11Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![xiaojunjie](https://avatars.discourse-cdn.com/v4/letter/x/ecb155/32.png) [@xiaojunjie](https://discuss.huggingface.co/u/xiaojunjie)\ **Post date:** [August 15, 2023, 8:27am UTC](https://discuss.huggingface.co/t/how-to-run-trainer-py-with-megatron-lm-plugin/50804/1 "2023-08-15T08:27:11Z")
**src/transformers/trainer.py** , as I understand it, supports deepspeed but not megatron-lm. Is it right? When I tried to make it support megatron-lm, I encountered some problems： 1. When and where to load megatron-lm ckpt? A. function [\_inner\_training\_loop](https://github.com/huggingface/transformers/blob/7a94ea4c645f560c675cf8f91377b5a04fda3379/src/transformers/trainer.py#L1686) B. prepare\_model in **accelerate/utils/megatron\_lm.py** C. other good way 2. Only the loss of last\_pp\_rank is valid in Megatron-LM, while the Trainer takes the mean of ranks. When and where to deal with loss of megatron? A. function \_nested\_gather in **trainer.py** B. [GitHub - NVIDIA/Megatron-LM: Ongoing research training transformer models at scale](https://github.com/NVIDIA/Megatron-LM) C. other good ways 3. Trainer calls accelerate.prepare for model and dataloader respectively, which will initialize megatron-lm repeatedly. How to avoid this problem A. Modify initialization in Megatron-LM/megatron/global\_vars.py B. Modify function initialize in **accelerate/utils/megatron\_lm.py** C. other good ways 4. In order to support the given dataset with different seq-length for each batch, when and where to padding it to meet requirements of megatron. (All batches have same seq-length) A. Accelerate B. Megatron-LM C. other good ways