# DeepSpeed integration for HuggingFace Seq2SeqTrainingArguments **URL:** \ **Category:** DeepSpeed\ **Created:** [February 22, 2024, 10:39am UTC](https://discuss.huggingface.co/t/deepspeed-integration-for-huggingface-seq2seqtrainingarguments/74323 "2024-02-22T10:39:33Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![Cameron-oos](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cameron-oos/32/23127_2.png) [@Cameron-oos](https://discuss.huggingface.co/u/Cameron-oos)\ **Post date:** [February 22, 2024, 10:39am UTC](https://discuss.huggingface.co/t/deepspeed-integration-for-huggingface-seq2seqtrainingarguments/74323/1 "2024-02-22T10:39:33Z")
Hello. I’m doing a classic whisper fine-tune using Huggingface 🤗 libraries. my data is in good format and has already been mapped. I can use **whisper-base** , but I need to fine-tune on **whisper-large-v2**. **whisper-large-v2** model requires **~24GB** of GPU VRAM. I have access to 2 V100 Cards which have 16GB VRAM each, so that’s 32GB, but split between 2 GPUs hence I need the deepspeed integration for multiple GPU deployment. Here is my Seq2SeqTrainingArguments: training\_args = Seq2SeqTrainingArguments( output\_dir=‘/home/camerono/Cameron-oos/whisper-rxn-tuned’, # change to a repo name of your choice deepspeed=‘/home/camerono/whisper-finetuning/testEnvFinetuning/ds\_config.json’, per\_device\_train\_batch\_size=16, gradient\_accumulation\_steps=1, # increase by 2x for every 2x decrease in batch size learning\_rate=1e-5, warmup\_steps=500, max\_steps=4000, gradient\_checkpointing=True, fp16=True, # Change to false if using CPU only evaluation\_strategy=‘steps’, per\_device\_eval\_batch\_size=8, predict\_with\_generate=True, generation\_max\_length=225, save\_steps=1000, eval\_steps=1000, logging\_steps=25, report\_to=[“tensorboard”], load\_best\_model\_at\_end=True, metric\_for\_best\_model=“wer”, greater\_is\_better=False, push\_to\_hub=True ) And here is my ds\_config.json file contents: { “fp16”: { “enabled”: true }, “zero\_optimization”: { “stage”: 2, “allgather\_partitions”: true, “allgather\_bucket\_size”: 2e8, “overlap\_comm”: true, “reduce\_scatter”: true, “reduce\_bucket\_size”: 2e8, “contiguous\_gradients”: true }, “optimizer”: { “type”: “AdamW”, “params”: { “lr”: 1e-5, “betas”: [0.9, 0.999], “eps”: 1e-8, “weight\_decay”: 0.01 } }, “scheduler”: { “type”: “WarmupLR”, “params”: { “warmup\_min\_lr”: 0, “warmup\_max\_lr”: 1e-5, “warmup\_num\_steps”: 500 } } } I do not have a CUDA and Torch mismatch. Im using nvcc version 11.5 and torch 1.11. They are compatible as can be seen here: [PyTorch Release 21.12 - NVIDIA Docs](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_21-12.html). The error Im getting: [2024-02-22 10:32:12,164] [INFO] [comm.py:668:init\_distributed] Initializing TorchBackend in DeepSpeed with backend nccl Using /home/camerono/.cache/torch\_extensions/py310\_cu115 as PyTorch extensions root… Using /home/camerono/.cache/torch\_extensions/py310\_cu115 as PyTorch extensions root… Detected CUDA files, patching ldflags Emitting ninja build file /home/camerono/.cache/torch\_extensions/py310\_cu115/fused\_adam/build.ninja… Building extension module fused\_adam… Allowing ninja to set a default number of workers… (overridable by setting the environment variable MAX\_JOBS=N) [1/3] /usr/bin/nvcc -DTORCH\_EXTENSION\_NAME=fused\_adam -DTORCH\_API\_INCLUDE\_EXTENSION\_H -DPYBIND11\_COMPILER\_TYPE="_gcc" -DPYBIND11\_STDLIB="libstdcpp" -DPYBIND11\_BUILD\_ABI="cxxabi1011" -I/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/csrc/adam -isystem /home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/include -isystem /home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/include/TH -isystem /home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/include/THC -isystem /home/camerono/.conda/envs/train/include/python3.10 -D\_GLIBCXX\_USE\_CXX11\_ABI=0 -D\_\_CUDA\_NO\_HALF\_OPERATORS -D\_\_CUDA\_NO\_HALF\_CONVERSIONS_\_ -D\_\_CUDA\_NO\_BFLOAT16\_CONVERSIONS\_\_ -D\_\_CUDA\_NO\_HALF2\_OPERATORS\_\_ --expt-relaxed-constexpr -gencode=arch=compute\_70,code=compute\_70 -gencode=arch=compute\_70,code=sm\_70 --compiler-options ‘-fPIC’ -O3 -DVERSION\_GE\_1\_1 -DVERSION\_GE\_1\_3 -DVERSION\_GE\_1\_5 -lineinfo --use\_fast\_math -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_70,code=compute\_70 -std=c++14 -c /home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/multi\_tensor\_adam.cu -o multi\_tensor\_adam.cuda.o FAILED: multi\_tensor\_adam.cuda.o /usr/bin/nvcc -DTORCH\_EXTENSION\_NAME=fused\_adam -DTORCH\_API\_INCLUDE\_EXTENSION\_H -DPYBIND11\_COMPILER\_TYPE="_gcc" -DPYBIND11\_STDLIB="libstdcpp" -DPYBIND11\_BUILD\_ABI="cxxabi1011" -I/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/csrc/adam -isystem /home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/include -isystem /home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/include/TH -isystem /home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/include/THC -isystem /home/camerono/.conda/envs/train/include/python3.10 -D\_GLIBCXX\_USE\_CXX11\_ABI=0 -D\_\_CUDA\_NO\_HALF\_OPERATORS -D\_\_CUDA\_NO\_HALF\_CONVERSIONS_\_ -D\_\_CUDA\_NO\_BFLOAT16\_CONVERSIONS\_\_ -D\_\_CUDA\_NO\_HALF2\_OPERATORS\_\_ --expt-relaxed-constexpr -gencode=arch=compute\_70,code=compute\_70 -gencode=arch=compute\_70,code=sm\_70 --compiler-options ‘-fPIC’ -O3 -DVERSION\_GE\_1\_1 -DVERSION\_GE\_1\_3 -DVERSION\_GE\_1\_5 -lineinfo --use\_fast\_math -gencode=arch=compute\_70,code=sm\_70 -gencode=arch=compute\_70,code=compute\_70 -std=c++14 -c /home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/multi\_tensor\_adam.cu -o multi\_tensor\_adam.cuda.o /usr/include/c++/11/bits/std\_function.h:435:145: error: parameter packs not expanded with ‘…’: 435 | function(\_Functor&& \_\_f) | ^ /usr/include/c++/11/bits/std\_function.h:435:145: note: ‘\_ArgTypes’ /usr/include/c++/11/bits/std\_function.h:530:146: error: parameter packs not expanded with ‘…’: 530 | operator=(\_Functor&& \_\_f) | ^ /usr/include/c++/11/bits/std\_function.h:530:146: note: ‘\_ArgTypes’ [2/3] c++ -MMD -MF fused\_adam\_frontend.o.d -DTORCH\_EXTENSION\_NAME=fused\_adam -DTORCH\_API\_INCLUDE\_EXTENSION\_H -DPYBIND11\_COMPILER\_TYPE="\_gcc" -DPYBIND11\_STDLIB="\_libstdcpp" -DPYBIND11\_BUILD\_ABI="\_cxxabi1011" -I/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/csrc/adam -isystem /home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/include -isystem /home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/include/TH -isystem /home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/include/THC -isystem /home/camerono/.conda/envs/train/include/python3.10 -D\_GLIBCXX\_USE\_CXX11\_ABI=0 -fPIC -std=c++14 -O3 -std=c++17 -g -Wno-reorder -DVERSION\_GE\_1\_1 -DVERSION\_GE\_1\_3 -DVERSION\_GE\_1\_5 -c /home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/fused\_adam\_frontend.cpp -o fused\_adam\_frontend.o ninja: build stopped: subcommand failed. Traceback (most recent call last): File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/utils/cpp\_extension.py”, line 1740, in \_run\_ninja\_build subprocess.run( File “/home/camerono/.conda/envs/train/lib/python3.10/subprocess.py”, line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’]’ returned non-zero exit status 1. The above exception was the direct cause of the following exception: Traceback (most recent call last): File “/home/camerono/whisper-finetuning/testEnvFinetuning/testingEnvTwo.py”, line 161, in trainer.train() File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/transformers/trainer.py”, line 1530, in train return inner\_training\_loop( File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/transformers/trainer.py”, line 1690, in \_inner\_training\_loop model, self.optimizer, self.lr\_scheduler = self.accelerator.prepare( File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/accelerate/accelerator.py”, line 1220, in prepare result = self.\_prepare\_deepspeed(\*args) File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/accelerate/accelerator.py”, line 1605, in \_prepare\_deepspeed engine, optimizer, \_, lr\_scheduler = deepspeed.initialize(\*\*kwargs) File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ **init**.py”, line 176, in initialize engine = DeepSpeedEngine(args=args, File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/runtime/engine.py”, line 307, in **init** self.\_configure\_optimizer(optimizer, model\_parameters) File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/runtime/engine.py”, line 1231, in \_configure\_optimizer basic\_optimizer = self.\_configure\_basic\_optimizer(model\_parameters) File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/runtime/engine.py”, line 1308, in \_configure\_basic\_optimizer optimizer = FusedAdam( File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/adam/fused\_adam.py”, line 94, in **init** fused\_adam\_cuda = FusedAdamBuilder().load() File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/op\_builder/builder.py”, line 478, in load return self.jit\_load(verbose) File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/op\_builder/builder.py”, line 522, in jit\_load op\_module = load(name=self.name, File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/utils/cpp\_extension.py”, line 1144, in load return \_jit\_compile( File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/utils/cpp\_extension.py”, line 1357, in \_jit\_compile \_write\_ninja\_file\_and\_build\_library( File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/utils/cpp\_extension.py”, line 1469, in \_write\_ninja\_file\_and\_build\_library \_run\_ninja\_build( File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/utils/cpp\_extension.py”, line 1756, in \_run\_ninja\_build raise RuntimeError(message) from e RuntimeError: Error building extension ‘fused\_adam’ Loading extension module fused\_adam… Traceback (most recent call last): File “/home/camerono/whisper-finetuning/testEnvFinetuning/testingEnvTwo.py”, line 161, in trainer.train() File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/transformers/trainer.py”, line 1530, in train return inner\_training\_loop( File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/transformers/trainer.py”, line 1690, in \_inner\_training\_loop model, self.optimizer, self.lr\_scheduler = self.accelerator.prepare( File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/accelerate/accelerator.py”, line 1220, in prepare result = self.\_prepare\_deepspeed(\*args) File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/accelerate/accelerator.py”, line 1605, in \_prepare\_deepspeed engine, optimizer, \_, lr\_scheduler = deepspeed.initialize(\*\*kwargs) File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ **init**.py”, line 176, in initialize engine = DeepSpeedEngine(args=args, File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/runtime/engine.py”, line 307, in **init** self.\_configure\_optimizer(optimizer, model\_parameters) File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/runtime/engine.py”, line 1231, in \_configure\_optimizer basic\_optimizer = self.\_configure\_basic\_optimizer(model\_parameters) File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/runtime/engine.py”, line 1308, in \_configure\_basic\_optimizer optimizer = FusedAdam( File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/adam/fused\_adam.py”, line 94, in **init** fused\_adam\_cuda = FusedAdamBuilder().load() File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/op\_builder/builder.py”, line 478, in load return self.jit\_load(verbose) File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/deepspeed/ops/op\_builder/builder.py”, line 522, in jit\_load op\_module = load(name=self.name, File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/utils/cpp\_extension.py”, line 1144, in load return \_jit\_compile( File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/utils/cpp\_extension.py”, line 1382, in \_jit\_compile return \_import\_module\_from\_library(name, build\_directory, is\_python\_module) File “/home/camerono/.conda/envs/train/lib/python3.10/site-packages/torch/utils/cpp\_extension.py”, line 1775, in \_import\_module\_from\_library module = importlib.util.module\_from\_spec(spec) File “”, line 571, in module\_from\_spec File “”, line 1176, in create\_module File “”, line 241, in \_call\_with\_frames\_removed ImportError: /home/camerono/.cache/torch\_extensions/py310\_cu115/fused\_adam/fused\_adam.so: cannot open shared object file: No such file or directory [2024-02-22 10:32:38,157] [INFO] [launch.py:316:sigkill\_handler] Killing subprocess 83715 [2024-02-22 10:32:38,162] [INFO] [launch.py:316:sigkill\_handler] Killing subprocess 83716 [2024-02-22 10:32:38,163] [ERROR] [launch.py:322:sigkill\_handler] [‘/home/camerono/.conda/envs/train/bin/python’, ‘-u’, ‘testingEnvTwo.py’, ‘–local\_rank=1’] exits with return code = 1