# LM example run\_clm.py isn't distributing data across multiple GPUs as expected **URL:** \ **Category:** šŸ¤—Transformers\ **Created:** [January 16, 2021, 10:38pm UTC](https://discuss.huggingface.co/t/lm-example-run-clm-py-isnt-distributing-data-across-multiple-gpus-as-expected/3239 "2021-01-16T22:38:50Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![saxon](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saxon/32/16199_2.png) [@saxon](https://discuss.huggingface.co/u/saxon)\ **Post date:** [January 16, 2021, 10:38pm UTC](https://discuss.huggingface.co/t/lm-example-run-clm-py-isnt-distributing-data-across-multiple-gpus-as-expected/3239/1 "2021-01-16T22:38:50Z")
**EDIT: I think the missing piece is the `-m torch.distributed.launch` flag from the terminal command. I will test this when I get a chance and update the thread if thatā€™s the fix.** I am fine-tuning GPT-2 using examples/language-modeling/run\_clm.py. It seems like the Trainer class instantiated in it will by default wrap the model in Distributed Data Parallel and spread it across the 4 gpus that I am providing it when I include CUDA\_VISIBLE\_DEVICES=0,1,2,3 at call time. However, when I run nvidia-smi only gpu:0 is being used. The first line the script prints is ``` 01/16/2021 02:39:40 - WARNING - __main__ - Process rank: -1, device: cuda:0, n_gpu: 0 distributed training: False, 16-bits training: False ``` I understand that for the model using DDP, n\_gpu should be 1? I think the print of n\_gpu=0 is just a result of n\_gpu not actually being available as a flag for configuration. The next line I get is ``` 01/16/2021 02:39:40 - INFO - __main__ - Training/evaluation parameters Trainin gArguments(output_dir=/data/saxon/nlp_abs_full/test, overwrite_output_dir=True, do_train=True, do_eval=False, do_predict=False, evaluation_strategy=EvaluationSt rategy.NO, prediction_loss_only=False, per_device_train_batch_size=5, per_device _eval_batch_size=8, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_e psilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, lr_schedule r_type=SchedulerType.LINEAR, warmup_steps=0, logging_dir=runs/Jan16_02-39-40_and rew.cs, logging_first_step=False, logging_steps=500, save_steps=500, save_total_ limit=3, no_cuda=False, seed=42, fp16=False, fp16_opt_level=O1, fp16_backend=aut o, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, data loader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name=/data/saxon/nlp_abs_full/test, disable_tqdm=False, remove_unused_colum ns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=N one, greater_is_better=None, ignore_data_skip=False, sharded_ddp=False, deepspee d=None, label_smoothing_factor=0.0, adafactor=False, _n_gpu=4) ``` It seems like whatā€™s happening here is somewhere in the process of setting up the trainer n\_gpu is defaulting to 4 (the number that I have), which I think is somehow interrupting the process that is supposed to happen where the trainer wraps the model in DDP. To try fixing this I added the line trainer.\_n\_gpu = 1 to force the value to the argument that it should be for [DDP according to the documentation](https://huggingface.co/transformers/main_classes/trainer.html#transformers.Trainer) However, this does not fix the problem, so Iā€™m stuck. I think this might be a bug in the example script, because the expected behavior, if I understand right, is that training should automatically use DDP when more than 1 gpu is available. Am I doing something wrong? The full command Iā€™m running is `TRANSFORMERS_CACHE=/data/saxon/cache CUDA_VISIBLE_DEVICES=0,1,2,3 python run_clm.py --model_name_or_path gpt2 --train_file /data/saxon/nlp_abs_full/ffl.txt --do_train --output_dir /data/saxon/nlp_abs_full/test --per_device_train_batch_size 5 --cache_dir /data/saxon/cache --save_total_limit 3 --save_steps 500 --overwrite_output_dir` ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [January 19, 2021, 9:10pm UTC](https://discuss.huggingface.co/t/lm-example-run-clm-py-isnt-distributing-data-across-multiple-gpus-as-expected/3239/2 "2021-01-19T21:10:12Z")
If you want to use DDP (distributed data parallel) you do need to launch the script with `python -m torch.distributed.launch`. ---
**Author:** ![sadra](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@sadra](https://discuss.huggingface.co/u/sadra)\ **Post date:** [March 3, 2022, 11:52pm UTC](https://discuss.huggingface.co/t/lm-example-run-clm-py-isnt-distributing-data-across-multiple-gpus-as-expected/3239/3 "2022-03-03T23:52:14Z")
@sgugger Just to clarify one thing, when launching script with `python -m torch.distributed.launch --nproc_per_node=8 script.py`. it will be a DDP training, and there is no need to set `n_gpu.` It became confusing to me, as it was logging warning you are setting n\_gpu to 1 per node. ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [March 4, 2022, 2:27pm UTC](https://discuss.huggingface.co/t/lm-example-run-clm-py-isnt-distributing-data-across-multiple-gpus-as-expected/3239/4 "2022-03-04T14:27:34Z")
There is no `n_gpu` argument you can set, so Iā€™m confused about your question. ---
**Author:** ![sadra](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@sadra](https://discuss.huggingface.co/u/sadra)\ **Post date:** [March 10, 2022, 8:37pm UTC](https://discuss.huggingface.co/t/lm-example-run-clm-py-isnt-distributing-data-across-multiple-gpus-as-expected/3239/5 "2022-03-10T20:37:20Z")
[this line](https://github.com/huggingface/transformers/blob/6b09328368324d170504c14bcd202856d0f851a3/examples/pytorch/translation/run_translation.py#L262) gave me `n_gpu` is 1 while I am using ddp with 8 GPU. So, I was confused. As you said there is no `n_gpu` to set, probably this warning is normal. ---
**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\ **Post date:** [August 17, 2022, 3:03pm UTC](https://discuss.huggingface.co/t/lm-example-run-clm-py-isnt-distributing-data-across-multiple-gpus-as-expected/3239/6 "2022-08-17T15:03:33Z")
does this solve your question: [Using Transformers with DistributedDataParallel ā€” any examples?](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775) ---
**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\ **Post date:** [May 17, 2023, 5:18am UTC](https://discuss.huggingface.co/t/lm-example-run-clm-py-isnt-distributing-data-across-multiple-gpus-as-expected/3239/7 "2023-05-17T05:18:11Z")
Hi @saxon , I hope you are well. sorry, I want to use the model for fine tunning the gpt2. my question is that how you send the data to the model, what is ff.txt and how it is organized? is it the whole data that you used I mean both training and validation? my dataset is two csv files that contain training and validation sentences.I would appreciate if you please help me to know how I can pass the data. many thanks ---
**Author:** ![saxon](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saxon/32/16199_2.png) [@saxon](https://discuss.huggingface.co/u/saxon)\ **Post date:** [May 17, 2023, 6:30am UTC](https://discuss.huggingface.co/t/lm-example-run-clm-py-isnt-distributing-data-across-multiple-gpus-as-expected/3239/8 "2023-05-17T06:30:03Z")
Itā€™s been a few years since I did this project but I believe ffl.txt was just a plaintext file containing natural text that I was fine-tuning GPT2 to generate. This is just the training text and I believe it can be formatted arbitrarily. There might be flags for setting test, etc, try checking the documentation for run\_clm.py and looking at the options [transformers/run\_clm.py at main Ā· huggingface/transformers Ā· GitHub](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py) ---
**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\ **Post date:** [May 17, 2023, 6:33am UTC](https://discuss.huggingface.co/t/lm-example-run-clm-py-isnt-distributing-data-across-multiple-gpus-as-expected/3239/9 "2023-05-17T06:33:46Z")
@saxon , thanks for your answer. you means that it is a txt file including the sentences each line one sample? I read the code it is a bit confusing fo rme ā˜¹ ---
**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\ **Post date:** [May 17, 2023, 6:35am UTC](https://discuss.huggingface.co/t/lm-example-run-clm-py-isnt-distributing-data-across-multiple-gpus-as-expected/3239/10 "2023-05-17T06:35:41Z")
@saxon , can i use it for GPT-Neo as well? I think it is a general code ---
**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\ **Post date:** [May 17, 2023, 6:48am UTC](https://discuss.huggingface.co/t/lm-example-run-clm-py-isnt-distributing-data-across-multiple-gpus-as-expected/3239/11 "2023-05-17T06:48:20Z")
HI @sgugger , I hope you are well. I want to use run\_clm.py code, is it all right to run it in this way. my target model is gpt-neo. and about the data is it right to use all data in a txt file, each row one sentence? should each sentence separated by ā€œ.ā€ at the end? or no ā€œ.ā€ needed at the end. CUDA\_VISIBLE\_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc\_per\_node 2 run\_clm.py --model\_name\_or\_path ./nlp/gpt\_neo/ --train\_file /nlp\_abs\_full/ffl.txt --do\_train --output\_dir /data/nlp\_abs\_full/test ā€“per\_device\_train\_batch\_size 5 --save\_total\_limit 3 --save\_steps 500