# Detecting single gpu within each node **URL:** \ **Category:** 🤗Accelerate\ **Created:** [January 16, 2023, 7:43pm UTC](https://discuss.huggingface.co/t/detecting-single-gpu-within-each-node/29828 "2023-01-16T19:43:42Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![mderakhshani](https://avatars.discourse-cdn.com/v4/letter/m/7ea924/32.png) [@mderakhshani](https://discuss.huggingface.co/u/mderakhshani)\ **Post date:** [January 16, 2023, 7:43pm UTC](https://discuss.huggingface.co/t/detecting-single-gpu-within-each-node/29828/1 "2023-01-16T19:43:42Z")
Hi, I have two nodes, each containing 3 A6000 GPUs. I am using Accelerate library to do multi-node training with two following config files: **1. default\_config\_1.yaml** ```auto command_file: null commands: null compute_environment: LOCAL_MACHINE deepspeed_config: deepspeed_multinode_launcher: standard gradient_accumulation_steps: 4 offload_optimizer_device: cpu offload_param_device: cpu zero3_init_flag: false zero_stage: 2 distributed_type: DEEPSPEED downcast_bf16: 'no' dynamo_backend: 'NO' fsdp_config: {} gpu_ids: null machine_rank: 0 main_process_ip: 10.149.4.12 main_process_port: 5000 main_training_function: main megatron_lm_config: {} mixed_precision: fp16 num_machines: 2 num_processes: 3 rdzv_backend: static same_network: true tpu_name: null tpu_zone: null use_cpu: false ``` **2. default\_config\_2.yaml** ```auto command_file: null commands: null compute_environment: LOCAL_MACHINE deepspeed_config: deepspeed_multinode_launcher: standard gradient_accumulation_steps: 4 offload_optimizer_device: cpu offload_param_device: cpu zero3_init_flag: false zero_stage: 2 distributed_type: DEEPSPEED downcast_bf16: 'no' dynamo_backend: 'NO' fsdp_config: {} gpu_ids: null machine_rank: 1 main_process_ip: 10.149.4.12 main_process_port: 5000 main_training_function: main megatron_lm_config: {} mixed_precision: fp16 num_machines: 2 num_processes: 3 rdzv_backend: static same_network: true tpu_name: null tpu_zone: null use_cpu: false ``` Using these YAML files, I will call the training script on each node separately as following: **Node1:** `accelerate launch --config_file=/home/username/.cache/huggingface/accelerate/default_config_1.yaml --multi_gpu train.py --data ./train.pkl --out_dir ./checkpoint/ --bs 80 --prefix_dim 1024 --prefix cc --prefix_length 5` **Node2:** `accelerate launch --config_file=/home/username/.cache/huggingface/accelerate/default_config_2.yaml --multi_gpu train.py --data ./train.pkl --out_dir ./checkpoint/ --bs 80 --prefix_dim 1024 --prefix cc --prefix_length 5` However, I can observe that only the Accelerate library uses the first GPU in each node. I have also tried without the deepspeed option, but I have the same issue. Could you please help me in this regard? Thanks. ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\ **Post date:** [January 17, 2023, 1:29pm UTC](https://discuss.huggingface.co/t/detecting-single-gpu-within-each-node/29828/2 "2023-01-17T13:29:08Z")
> [@mderakhshani](#): > > However, I can observe that only the Accelerate library uses the first GPU in each node. I have also tried without the deepspeed option, but I have the same issue. Could you please help me in this regard? Try just with: ```bash accelerate launch --config_file=/home/username/.cache/huggingface/accelerate/default_config_1.yaml train.py --data ./train.pkl --out_dir ./checkpoint/ --bs 80 --prefix_dim 1024 --prefix cc --prefix_length 5 ``` Or, let the config file do all the work ---
**Author:** ![mderakhshani](https://avatars.discourse-cdn.com/v4/letter/m/7ea924/32.png) [@mderakhshani](https://discuss.huggingface.co/u/mderakhshani)\ **Post date:** [January 17, 2023, 2:57pm UTC](https://discuss.huggingface.co/t/detecting-single-gpu-within-each-node/29828/3 "2023-01-17T14:57:35Z")
Thanks for the reply. It still does not work. Following, I will provide my configs again with test command: - node 1 (master): ```auto compute_environment: LOCAL_MACHINE deepspeed_config: {} distributed_type: MULTI_GPU downcast_bf16: 'no' dynamo_backend: 'NO' fsdp_config: {} gpu_ids: all machine_rank: 0 main_process_ip: 10.149.4.12 main_process_port: 12345 main_training_function: main megatron_lm_config: {} mixed_precision: fp16 num_machines: 2 num_processes: 3 rdzv_backend: static same_network: true use_cpu: false ``` - node 2 (slave) ```auto compute_environment: LOCAL_MACHINE deepspeed_config: {} distributed_type: MULTI_GPU downcast_bf16: 'no' dynamo_backend: 'NO' fsdp_config: {} gpu_ids: all machine_rank: 1 main_process_ip: 10.149.4.12 main_process_port: 12345 main_training_function: main megatron_lm_config: {} mixed_precision: fp16 num_machines: 2 num_processes: 3 rdzv_backend: static same_network: true use_cpu: false ``` I ran the following code to check if the nodes can find each other or not by running the following commands: - node 1 (master) Command: ```auto accelerate-launch --config_file=/home/username/.cache/huggingface/accelerate/default_config_1.yaml /home/mderaksh/miniconda3/envs/dassl/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_script.py ``` output: ```auto $ accelerate-launch --config_file=/home/username/.cache/huggingface/accelerate/default_config_1.yaml /home/mderaksh/miniconda3/envs/dassl/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_script.py **Initialization** Testing, testing. 1, 2, 3. Distributed environment: MULTI_GPU Backend: nccl Num processes: 2 Process index: 0 Local process index: 0 Device: cuda:0 Mixed precision type: fp16 **Test random number generator synchronization** All rng are properly synched. **DataLoader integration test** 0 tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], device='cuda:0') Non-shuffled dataloader passing. Shuffled dataloader passing. Non-shuffled central dataloader passing. Shuffled central dataloader passing. **Training integration test** Model dtype: torch.float32, torch.float32. Input dtype: torch.float32 Model dtype: torch.float32, torch.float32. Input dtype: torch.float32 Training yielded the same results on one CPU or distributed setup with no batch split. Model dtype: torch.float32, torch.float32. Input dtype: torch.float32 Training yielded the same results on one CPU or distributes setup with batch split. FP16 training check. Model dtype: torch.float32, torch.float32. Input dtype: torch.float32 BF16 training check. Model dtype: torch.float32, torch.float32. Input dtype: torch.float32 ``` - node 2 (slave): Command: ```auto $ accelerate-launch --config_file=/home/username/.cache/huggingface/accelerate/default_config_2.yaml /home/mderaksh/miniconda3/envs/dassl/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_script.py ``` Output: ```auto **Initialization** Testing, testing. 1, 2, 3. Distributed environment: MULTI_GPU Backend: nccl Num processes: 2 Process index: 1 Local process index: 0 Device: cuda:0 Mixed precision type: fp16 **Test random number generator synchronization** All rng are properly synched. **DataLoader integration test** 1 tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], device='cuda:0') Shuffled dataloader passing. Shuffled central dataloader passing. **Training integration test** Model dtype: torch.float32, torch.float32. Input dtype: torch.float32 Model dtype: torch.float32, torch.float32. Input dtype: torch.float32 Training yielded the same results on one CPU or distributed setup with no batch split. Model dtype: torch.float32, torch.float32. Input dtype: torch.float32 Training yielded the same results on one CPU or distributes setup with batch split. FP16 training check. Model dtype: torch.float32, torch.float32. Input dtype: torch.float32 BF16 training check. Model dtype: torch.float32, torch.float32. Input dtype: torch.float32 ``` However, I cannot figure out what the problem is. ☹ **Update:** I tried the following link to do distributed learning on two nodes using `torch.distributed.launch`, and it seems it is working. However, Accelerate does not. > **[PyTorch Distributed Training](https://leimao.github.io/blog/PyTorch-Distributed-Training/)** > > PyTorch Distributed Training for Dummies