# Distributed Training run\_summarization.py

**URL:** <https://discuss.huggingface.co/t/distributed-training-run-summarization-py/8789>\
**Category:** Amazon SageMaker\
**Created:** [July 29, 2021, 5:02am UTC](https://discuss.huggingface.co/t/distributed-training-run-summarization-py/8789 "2021-07-29T05:02:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![cdwyer1bod](https://avatars.discourse-cdn.com/v4/letter/c/5daacb/32.png) [@cdwyer1bod](https://discuss.huggingface.co/u/cdwyer1bod)\
**Post date:** [July 29, 2021, 5:02am UTC](https://discuss.huggingface.co/t/distributed-training-run-summarization-py/8789/1 "2021-07-29T05:02:34Z")

</div>

Hi,  
I cannot for the life of me figure out what is going wrong. I am following the tutorial 08 distributed training and when sending the job to AWS the error continually shows up. I am running it on a local jupyter notebook with these setups:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/6/6768326add5e5c2d823b57f64f80e39d3a9c1816.png)  
…  
[1,0]:storing [https://huggingface.co/facebook/bart-large-cnn/resolve/main/pytorch\_model.bin](https://huggingface.co/facebook/bart-large-cnn/resolve/main/pytorch_model.bin) in cache at /root/.cache/huggingface/transformers/4ccdf4cdc01b790f9f9c636c7695b5d443180e8dbd0cbe49e07aa918dda1cef0.fa29468c10a34ef7f6cfceba3b174d3ccc95f8d755c3ca1b829aff41cc92a300  
[1,0]:creating metadata file for /root/.cache/huggingface/transformers/4ccdf4cdc01b790f9f9c636c7695b5d443180e8dbd0cbe49e07aa918dda1cef0.fa29468c10a34ef7f6cfceba3b174d3ccc95f8d755c3ca1b829aff41cc92a300  
[1,5]:Environment variable SAGEMAKER\_INSTANCE\_TYPE is not set  
[1,7]:Environment variable SAGEMAKER\_INSTANCE\_TYPE is not set  
[1,3]:Environment variable SAGEMAKER\_INSTANCE\_TYPE is not set  
[1,0]:Environment variable SAGEMAKER\_INSTANCE\_TYPE is not set  
[1,1]:Environment variable SAGEMAKER\_INSTANCE\_TYPE is not set  
[1,4]:Environment variable SAGEMAKER\_INSTANCE\_TYPE is not set  
[1,6]:Environment variable SAGEMAKER\_INSTANCE\_TYPE is not set

…  
[1,6]:#015100%|██████████| 1/1 [00:00\<00:00, 8.65ba/s][1,6]:#015100%|██████████| 1/1 [00:00\<00:00, 8.62ba/s]  
[1,7]:#015Downloading: 0%| | 0.00/2.17k [00:00\<?, ?B/s][1,7]:#015Downloading: 5.61kB [00:00, 5.02MB/s]  
[1,0]:#015Downloading: 0%| | 0.00/2.17k [00:00\<?, ?B/s][1,0]:#015Downloading: 5.61kB [00:00, 4.54MB/s]  
[1,0]:Using amp fp16 backend  
[1,0]:\*\*\*\*\* Running training \*\*\*\*\*  
[1,0]: Num examples = 14732  
[1,0]: Num Epochs = 3  
[1,0]: Instantaneous batch size per device = 4  
[1,0]: Total train batch size (w. parallel, distributed & accumulation) = 32  
[1,0]: Gradient Accumulation steps = 1  
[1,0]: Total optimization steps = 1383  
[1,0]:#015 0%| | 0/1383 [00:00\<?, ?it/s][1,0]:#015 0%| | 1/1383 [00:02\<59:19, 2.58s/it][1,0]:#015 0%| | 2/1383 [00:04\<51:42, 2.25s/it][1,0]:#015 0%| | 3/1383 [00:05\<45:46, 1.99s/it][1,0]:#015 0%| | 4/1383 [00:07\<45:13, 1.97s/it][1,0]:#015 0%| | 5/1383 [00:09\<43:12, 1.88s/it][1,0]:#015 0%| | 6/1383 [00:10\<41:53, 1.83s/it][1,0]:#015 1%| | 7/1383 [00:12\<41:48, 1.82s/it][1,0]:#015 1%| | 8/1383 [00:14\<43:16, 1.89s/it][1,0]:#015 1%| | 9/1383 [00:16\<44:01, 1.92s/it][1,0]:#015 1%| | 10/1383 [00:18\<44:03, 1.93s/it][1,0]:#015 1%| | 11/1383 [00:20\<44:13, 1.93s/it][1,0]:#015 1%| | 12/1383 [00:22\<44:31, 1.95s/it][1,0]:#015 1%| | 13/1383 [00:24\<44:06, 1.93s/it][1,0]:#015 1%| | 14/1383 [00:26\<43:58, 1.93s/it][1,0]:#015 1%| | 15/1383 [00:28\<43:42, 1.92s/it][1,0]:#015 1%| | 16/1383 [00:30\<43:46, 1.92s/it][1,0]:#015 1%| | 17/1383 [00:32\<43:47, 1.92s/it]--------------------------------------------------------------------------

MPI\_ABORT was invoked on rank 1 in communicator MPI COMMUNICATOR 5 DUP FROM 0  
with errorcode 1.

## NOTE: invoking MPI\_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them.

2021-07-29 04:41:50 Uploading - Uploading generated training model  
2021-07-29 04:41:50 Failed - Training job failed

…

UnexpectedStatusException: Error for Training job huggingface-pytorch-training-2021-07-29-04-30-08-439: Failed. Reason: AlgorithmError: ExecuteUserScriptError:  
Command “mpirun --host algo-1 -np 8 --allow-run-as-root --tag-output --oversubscribe -mca btl\_tcp\_if\_include eth0 -mca oob\_tcp\_if\_include eth0 -mca plm\_rsh\_no\_tree\_spawn 1 -mca pml ob1 -mca btl ^openib -mca orte\_abort\_on\_non\_zero\_status 1 -mca btl\_vader\_single\_copy\_mechanism none -mca plm\_rsh\_num\_concurrent 1 -x NCCL\_SOCKET\_IFNAME=eth0 -x NCCL\_DEBUG=INFO -x LD\_LIBRARY\_PATH -x PATH -x SMDATAPARALLEL\_USE\_SINGLENODE=1 -x FI\_PROVIDER=efa -x RDMAV\_FORK\_SAFE=1 -x LD\_PRELOAD=/opt/conda/lib/python3.6/site-packages/gethostname.cpython-36m-x86\_64-linux-gnu.so smddprun /opt/conda/bin/python3.6 -m mpi4py run\_summarization.py --dataset\_name samsum --do\_eval True --do\_predict True --do\_train True --fp16 True --learning\_rate 5e-05 --model\_name\_or\_path facebook/bart-large-cnn --num\_train\_epochs 3 --output\_dir /opt/ml/model --per\_device\_eval\_batch\_size 4 --per\_device\_train\_batch\_size 4 --predict\_with\_generate True --seed 7”  
[1,2]:Environment variable SAGEMAKER\_INSTANCE\_TYPE is n

Any help would be greatly appreciated. The CloudWatch logs don’t really have anything else to say. Definitely seems like a problem with setting the environment variable SAGEMAKER\_INSTANCE\_TYPE but I thought I already set it by specifying instance\_type when initializing huggingface\_estimator??

Thanks 🙂

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [July 29, 2021, 6:45am UTC](https://discuss.huggingface.co/t/distributed-training-run-summarization-py/8789/2 "2021-07-29T06:45:56Z")

</div>

Hey @cdwyer1bod,

Thanks for opening the thread. Happy to help you.  
Could still share the full cloudwatch logs? sometimes the errors are a bit hidden.

I saw you changed the instance `ml.p3dn.24xlarge` to `ml.p3.16xlarge` and kept the batch\_size this could be the issue. Could reduce the batch\_size to 2 or change the instances type?

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [July 29, 2021, 7:15am UTC](https://discuss.huggingface.co/t/distributed-training-run-summarization-py/8789/3 "2021-07-29T07:15:59Z")

</div>

For me it worked with `ml.p3.16xlarge` and batch\_size of 2

---

<div class="post-metadata">

**Author:** ![cdwyer1bod](https://avatars.discourse-cdn.com/v4/letter/c/5daacb/32.png) [@cdwyer1bod](https://discuss.huggingface.co/u/cdwyer1bod)\
**Post date:** [July 30, 2021, 4:31am UTC](https://discuss.huggingface.co/t/distributed-training-run-summarization-py/8789/4 "2021-07-30T04:31:13Z")

</div>

Hey @philschmid thanks for the solution,

It ended up being a CUDA memory issue, I bumped down the learning rate, upped the instance number to 2 and put batch\_size at 2 and it worked after a while.

Thanks 🙂