# "No space left on device" when using HuggingFace + SageMaker **URL:** \ **Category:** Amazon SageMaker\ **Created:** [April 10, 2021, 12:07pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406 "2021-04-10T12:07:55Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![LeoCordoba](https://avatars.discourse-cdn.com/v4/letter/l/da6949/32.png) [@LeoCordoba](https://discuss.huggingface.co/u/LeoCordoba)\ **Post date:** [April 10, 2021, 12:07pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/1 "2021-04-10T12:07:55Z")
Hi, I’m trying to train a model using a HuggingFace estimator in SageMaker but I keep getting this error after a few minutes: [1,15]: File “pyarrow/ipc.pxi”, line 365, in pyarrow.lib.\_CRecordBatchWriter.write\_batch [1,15]: File “pyarrow/error.pxi”, line 97, in pyarrow.lib.check\_status [1,15]:OSError: [Errno 28] Error writing bytes to file. Detail: [errno 28] No space left on device [1,15]: I’m not sure what is triggering this problem because the volume size is high (volume\_size=1024) My hyperparameters are: {‘per\_device\_train\_batch\_size’: 4, ‘per\_device\_eval\_batch\_size’: 4, ‘model\_name\_or\_path’: ‘google/mt5-small’, ‘dataset\_name’: ‘mlsum’, ‘dataset\_config’: ‘es’, ‘text\_column’: ‘text’, ‘summary\_column’: ‘summary’, ‘max\_target\_length’: 64, ‘do\_train’: True, ‘do\_eval’: True, ‘do\_predict’: True, ‘predict\_with\_generate’: True, ‘output\_dir’: ‘/opt/ml/model’, ‘num\_train\_epochs’: 3, ‘seed’: 7, ‘fp16’: True, ‘save\_strategy’: ‘no’} And my estimator is: # create the Estimator huggingface\_estimator = HuggingFace( entry\_point=‘run\_summarization.py’, # script source\_dir=‘./examples/seq2seq’, # relative path to example git\_config=git\_config, instance\_type=‘ml.p3.16xlarge’, instance\_count=2, volume\_size=1024, transformers\_version=‘4.4.2’, pytorch\_version=‘1.6.0’, py\_version=‘py36’, role=role, hyperparameters = hyperparameters, distribution = distribution ) Any help would be very much appreciated! * * * Some more details: - I’m calling fit without extra params, just like this: huggingface\_estimator.fit() - The entry point is this public script: [https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run\_summarization.py](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_summarization.py) - From traceback I see that the error is happening on line 433: load\_from\_cache\_file=not data\_args.overwrite\_cache, (I guess something is happening here but not totally sure what) - At the moment I’m not saving checkpoints (to prevent that causing the error), using the param ‘save\_strategy’: ‘no’ - The dataset isn’t that big, 1.7 GB. - The model is quite big, but less than 3 GB - My volume is 1024 GB ---
**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\ **Post date:** [April 10, 2021, 12:42pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/2 "2021-04-10T12:42:59Z")
Could you also include your `.fit()` call so that the example can be reproduced? And a link to the `run_summarization.py` if public? Do you have a sense of what could take up a lot of storage? do you checkpoint a large model very frequently? or do you read a large dataset? ---
**Author:** ![LeoCordoba](https://avatars.discourse-cdn.com/v4/letter/l/da6949/32.png) [@LeoCordoba](https://discuss.huggingface.co/u/LeoCordoba)\ **Post date:** [April 10, 2021, 2:39pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/3 "2021-04-10T14:39:55Z")
Hi Olivier, thanks for your response! I’ve just edited my question including that information. Let me know if you need more data ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [April 12, 2021, 1:11pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/5 "2021-04-12T13:11:17Z")
Hey @LeoCordoba, Your error is coming from caching the dataset. Datasets is caching the dataset on disk to work with it properly. The default cache\_dir is `~/.cache/huggingface/datasets`. This directory seems not to be on the mounted EBS volume. To fix your problem you need to define a `cache_dir`in the `load_dataset` method. You can copy the `run_summarization.py` to your local filesystem and adjust line 313 from ```python datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name) ``` to ```python datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name,cache_dir="opt/ml/input") ``` > **[Loading methods](https://huggingface.co/docs/datasets/package_reference/loading_methods?highlight=cache_dir)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ---
**Author:** ![LeoCordoba](https://avatars.discourse-cdn.com/v4/letter/l/da6949/32.png) [@LeoCordoba](https://discuss.huggingface.co/u/LeoCordoba)\ **Post date:** [April 12, 2021, 5:58pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/6 "2021-04-12T17:58:51Z")
Thanks @philschmid ! That makes sense 👌 ---
**Author:** ![spranjal25](https://avatars.discourse-cdn.com/v4/letter/s/f0a364/32.png) [@spranjal25](https://discuss.huggingface.co/u/spranjal25)\ **Post date:** [February 15, 2022, 7:58am UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/7 "2022-02-15T07:58:31Z")
Hey @philschmid, Could you help me with a similar issue here, I am not using load\_dataset, instead I’m created Dataset from pandas through Dataset.from\_pandas. getting the same error: [Errno 28] No space left on device: ‘/home/ec2-user/.cache/huggingface’ I have sufficient storage on the instance and the dataset isn’t more than 20 GBs either. Do you have a possible solution to this as Dataset.from\_pandas doesn’t seem to have cache\_dir as an argument to it. Any help is appreciated. ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [February 15, 2022, 8:12am UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/8 "2022-02-15T08:12:32Z")
Hey @spranjal25, it is also possible to set the `cache_dir` via the ENVIRONMENT variable. > The default cache directory is `~/.cache/huggingface/datasets` . Change the cache location by setting the shell environment variable, `HF_DATASETS_CACHE` to another directory ```bash HF_DATASETS_CACHE="/path/to/another/directory" ``` Here you can find more documentation on this: [https://huggingface.co/docs/datasets/cache.html](https://huggingface.co/docs/datasets/cache.html) ---
**Author:** ![spranjal25](https://avatars.discourse-cdn.com/v4/letter/s/f0a364/32.png) [@spranjal25](https://discuss.huggingface.co/u/spranjal25)\ **Post date:** [February 15, 2022, 10:15am UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/9 "2022-02-15T10:15:39Z")
Thanks, that worked. @philschmid ---
**Author:** ![Vinayaks117](https://avatars.discourse-cdn.com/v4/letter/v/ebca7d/32.png) [@Vinayaks117](https://discuss.huggingface.co/u/Vinayaks117)\ **Post date:** [March 30, 2022, 5:29pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/10 "2022-03-30T17:29:01Z")
Hi @philschmid Greeting!! We are also experiencing “No space left on device” when training a BERT model using a HuggingFace estimator in SageMaker pipelines training job. Could you please help? Please let me know if you need any additional details. bert\_estimator = HuggingFace( entry\_point=“train.py”, source\_dir="./scripts", base\_job\_name=base\_job\_prefix + “/training”, instance\_type=“ml.p3.2xlarge”, instance\_count=1, volume\_size=1024, role=role, transformers\_version=“4.11.0”, pytorch\_version=“1.9.0”, py\_version=“py38”, hyperparameters=hyperparameters, sagemaker\_session=sagemaker\_session, ) Additional details: - Dataset size = 50 MB - The model is quite big, but less than 3 GB - My volume is 1024 GB - Model checkpoints are being saved in “/opt/ml/model/model\_02” directory Error details: #015 #033[ASaving model checkpoint to /model\_02/checkpoint-47866 Configuration saved in /model\_02/checkpoint-47866/config.json Model weights saved in /model\_02/checkpoint-47866/pytorch\_model.bin tokenizer config file saved in /model\_02/checkpoint-47866/tokenizer\_config.json Special tokens file saved in /model\_02/checkpoint-47866/special\_tokens\_map.json Traceback (most recent call last): File “/opt/conda/lib/python3.8/site-packages/torch/serialization.py”, line 379, in save \_save(obj, opened\_zipfile, pickle\_module, pickle\_protocol) File “/opt/conda/lib/python3.8/site-packages/torch/serialization.py”, line 499, in \_save zip\_file.write\_record(name, storage.data\_ptr(), num\_bytes) OSError: [Errno 28] No space left on device During handling of the above exception, another exception occurred: Traceback (most recent call last): File “train.py”, line 256, in trainer.train() File “/opt/conda/lib/python3.8/site-packages/transformers/trainer.py”, line 1383, in train self.\_maybe\_log\_save\_evaluate(tr\_loss, model, trial, epoch, ignore\_keys\_for\_eval) File “/opt/conda/lib/python3.8/site-packages/transformers/trainer.py”, line 1487, in \_maybe\_log\_save\_evaluate self.\_save\_checkpoint(model, trial, metrics=metrics) File “/opt/conda/lib/python3.8/site-packages/transformers/trainer.py”, line 1579, in \_save\_checkpoint torch.save(self.optimizer.state\_dict(), os.path.join(output\_dir, OPTIMIZER\_NAME)) File “/opt/conda/lib/python3.8/site-packages/torch/serialization.py”, line 380, in save return File “/opt/conda/lib/python3.8/site-packages/torch/serialization.py”, line 259, in **exit** self.file\_like.write\_end\_of\_file() RuntimeError: [enforce fail at inline\_container.cc:298] . unexpected pos 789763200 vs 789763088 terminate called after throwing an instance of ‘c10::Error’ what(): [enforce fail at inline\_container.cc:298] . unexpected pos 789763200 vs 789763088 Thank you. ---
**Author:** ![natgolovach](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/natgolovach/32/1818_2.png) [@natgolovach](https://discuss.huggingface.co/u/natgolovach)\ **Post date:** [March 31, 2022, 6:50pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/11 "2022-03-31T18:50:03Z")
I have the similar error while tokenizing the dataset. “[Errno 28] Error writing bytes to file. Detail: [errno 28] No space left on device” I use the original run\_mlm.py script to train MLM on my data using Sagemaker. I tried both estimators: HuggingFace and PyTorch. Same issue. I tried all the mentioned above solutions but it does not help. > #015Running tokenizer on dataset line\_by\_line #41: 24%|██▍ | 77/320 [01:45\<05:32, 1.37s/ba]multiprocess.pool.RemoteTraceback: “”"Traceback (most recent call last): > File “/opt/conda/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2354, in \_map\_single > writer.write\_batch(batch) > File “/opt/conda/lib/python3.6/site-packages/datasets/arrow\_writer.py”, line 496, in write\_batch > self.write\_table(pa\_table, writer\_batch\_size) > File “/opt/conda/lib/python3.6/site-packages/datasets/arrow\_writer.py”, line 513, in write\_table > self.pa\_writer.write\_batch(batch) > File “pyarrow/ipc.pxi”, line 408, in pyarrow.lib.\_CRecordBatchWriter.write\_batch > File “pyarrow/error.pxi”, line 112, in pyarrow.lib.check\_statusOSError: [Errno 28] Error writing bytes to file. Detail: [errno 28] No space left on deviceDuring handling of the above exception, another exception occurred:Traceback (most recent call last): > File “/opt/conda/lib/python3.6/site-packages/multiprocess/pool.py”, line 119, in worker > result = (True, func(\*args, \*\*kwds)) > File “/opt/conda/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 519, in wrapper#015Running tokenizer on dataset line\_by\_line #7: 24%|██▍ | 78/320 [01:45\<03:34, 1.13ba/s]#033[A#033[A#033[A#033[A#033[A#033[A#033[A > out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs) > File “/opt/conda/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 486, in wrapper > out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)#015Running tokenizer on dataset line\_by\_line #5: 27%|██▋ | 85/320 [01:45\<03:27, 1.13ba/s]#033[A#033[A#033[A#033[A#033[A > File “/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py”, line 458, in wrapper > out = func(self, \*args, \*\*kwargs) > File “/opt/conda/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 2360, in \_map\_single > writer.finalize() > File “/opt/conda/lib/python3.6/site-packages/datasets/arrow\_writer.py”, line 528, in finalize > self.pa\_writer.close() > File “pyarrow/ipc.pxi”, line 445, in pyarrow.lib.\_CRecordBatchWriter.close > File “pyarrow/error.pxi”, line 112, in pyarrow.lib.check\_statusOSError: [Errno 28] Error writing bytes to file. Detail: [errno 28] No space left on device ---
**Author:** ![natgolovach](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/natgolovach/32/1818_2.png) [@natgolovach](https://discuss.huggingface.co/u/natgolovach)\ **Post date:** [April 1, 2022, 5:10am UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/12 "2022-04-01T05:10:05Z")
I’ve solved it today. In HF’s [run\_mlm.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py) there is [datasets.map()](https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes#datasets.Dataset.map) function ([link to the line in code](https://github.com/huggingface/transformers/blob/5807054bd346367efb8d771605f6aa0a4e46ca5b/examples/pytorch/language-modeling/run_mlm.py#L408)) and I added argument to this function: ‘keep\_in\_memory=True’ to avoid caching tokenized dataset. ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [April 4, 2022, 7:23am UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/13 "2022-04-04T07:23:24Z")
Hello @Vinayaks117, could you please share your training script? The error is probably similar to the ones in the thread in that you need to adjust the `cache_dir` for `transformers` or `datasets`. ---
**Author:** ![Vinayaks117](https://avatars.discourse-cdn.com/v4/letter/v/ebca7d/32.png) [@Vinayaks117](https://discuss.huggingface.co/u/Vinayaks117)\ **Post date:** [April 6, 2022, 5:31pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/14 "2022-04-06T17:31:48Z")
Hi @philschmid As requested please find the training script in my github repo. I could not attach here hence I uploaded in one of my repos. > Thank you. ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [April 6, 2022, 6:06pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/15 "2022-04-06T18:06:33Z")
Thanks for providing! Coud you try to adjust the `cache` dir for `transformers` and `datasets` at the top of your script? ```python import os cache_dir = os.makedirs("cache",exist_ok=True) os.environ['TRANSFORMERS_CACHE'] = "cache" os.environ['HF_DATASETS_CACHE'] = "cache" ``` ---
**Author:** ![Vinayaks117](https://avatars.discourse-cdn.com/v4/letter/v/ebca7d/32.png) [@Vinayaks117](https://discuss.huggingface.co/u/Vinayaks117)\ **Post date:** [April 7, 2022, 6:39am UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/16 "2022-04-07T06:39:41Z")
Thanks for the details. If we look at the TrainingArguments in training script then we are saving checkpoints at the end of every epoch. save\_strategy=IntervalStrategy.EPOCH and output\_dir = ‘/opt/ml/model/model\_02’ Note: I observed this space issue because we are saving checkpoints at every epoch. So should we update cache\_dir or os.environ[‘TRANSFORMERS\_CACHE’] in TrainingArguments to store the checkpoints in cache\_dir? cache\_dir = os.makedirs(“cache”, exist\_ok=True) os.environ[‘TRANSFORMERS\_CACHE’] = “cache” os.environ[‘HF\_DATASETS\_CACHE’] = “cache” ---
**Author:** ![Vinayaks117](https://avatars.discourse-cdn.com/v4/letter/v/ebca7d/32.png) [@Vinayaks117](https://discuss.huggingface.co/u/Vinayaks117)\ **Post date:** [April 12, 2022, 4:19pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/17 "2022-04-12T16:19:19Z")
@philschmid Could you please clarify on above query? ---
**Author:** ![MaximusDecimusMeridi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maximusdecimusmeridi/32/3390_2.png) [@MaximusDecimusMeridi](https://discuss.huggingface.co/u/MaximusDecimusMeridi)\ **Post date:** [April 13, 2022, 2:39pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/18 "2022-04-13T14:39:01Z")
@philschmid I am running into a similar issue, but about halfway through my training. ``` OSError: [Errno 28] No space left on device ``` ```auto Traceback (most recent call last): File "train.py", line 182, in trainer.train() File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1328, in train self._maybe_log_save_evaluate(tr_loss, model, trial, epoch) File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1409, in _maybe_log_save_evaluate self._save_checkpoint(model, trial, metrics=metrics) File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1499, in _save_checkpoint torch.save(self.optimizer.state_dict(), os.path.join(output_dir, "optimizer.pt")) File "/opt/conda/lib/python3.6/site-packages/torch/serialization.py", line 373, in save return File "/opt/conda/lib/python3.6/site-packages/torch/serialization.py", line 259, in __exit__ self.file_like.write_end_of_file() ``` I have set cache\_dir=“opt/ml/input” in my load\_dataset. I have also assigned 100 GB additional volume. I also tried setting ```auto import os cache_dir = os.makedirs("cache",exist_ok=True) os.environ['TRANSFORMERS_CACHE'] = "cache" os.environ['HF_DATASETS_CACHE'] = "cache" ``` I’m also passing train\_volume\_size parameter to estimator: ```auto estimator = HuggingFace( entry_point = 'train.py', # fine-tuning script used in training jon source_dir = 'embed_source', # directory where fine-tuning script is stored instance_type = instance_type, # instances type used for the training job instance_count = 1, # the number of instances used for training role = get_execution_role(), # Iam role used in training job to access AWS ressources, transformers_version = '4.6', # the transformers version used in the training job pytorch_version = '1.7', # the pytorch_version version used in the training job py_version = 'py36', # the python version used in the training job hyperparameters = hyperparameters, # the hyperparameter used for running the training job metric_definitions = metric_definitions, # the metrics regex definitions to extract logs output_path=os.path.join(dataconnector.version_s3_prefix, "models"), code_location=os.path.join(dataconnector.version_s3_prefix, "models"), train_volume_size = 100 ) ``` But to no avail - fails after about 2 hours. Any ideas of what else I can do? ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [April 13, 2022, 3:15pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/19 "2022-04-13T15:15:06Z")
The parameter for the estimator to increase volume is in v2 `volume_size` [documentation](https://sagemaker.readthedocs.io/en/stable/v2.html?highlight=volume_size#renamed-estimator-parameters). Additionally, you could use `checkpointing`, which saves files which are saved to `/opt/ml/checkpoints` in sync to a s3 bucket defined in the `HuggingFace` estimator. [HF doc](https://huggingface.co/docs/sagemaker/train#spot-instances) [AWS Doc](https://docs.aws.amazon.com/sagemaker/latest/dg/model-checkpoints.html#model-checkpoints-enable) ---
**Author:** ![MaximusDecimusMeridi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maximusdecimusmeridi/32/3390_2.png) [@MaximusDecimusMeridi](https://discuss.huggingface.co/u/MaximusDecimusMeridi)\ **Post date:** [April 14, 2022, 3:23pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/20 "2022-04-14T15:23:02Z")
> [@philschmid](#): > > The parameter for the estimator to increase volume is in v2 `volume_size` Thanks, I updated the volume size and added checkpointing. It seems the job fails before I complete the first epoch though. My training data consists of 1.7M short text descriptions (~100 MB) and 23 classes. Would a distributed approach help here? Like in this post > **[Distributed fine-tuning of a BERT Large model for a Question-Answering Task...](https://aws.amazon.com/blogs/machine-learning/distributed-fine-tuning-of-a-bert-large-model-for-a-question-answering-task-using-hugging-face-transformers-on-amazon-sagemaker/)** > > From training new models to deploying them in production, Amazon SageMaker offers the most complete set of tools for startups and enterprises to harness the power of machine learning (ML) and Deep Learning. With its Transformers open-source library... ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [April 19, 2022, 6:45am UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/21 "2022-04-19T06:45:32Z")
You job fails with the following error? Or do you see something different? > [@MaximusDecimusMeridi](#): > > ```auto > OSError: [Errno 28] No space left on device > > ``` Distributed can help you either speed up you training or make it possible to fine-tune models which are not fit onto a single GPU. Since your corpus doesn’t sounds that big there is no need yet to go with distributed training [Next page](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406.md?page=2)