# Cannot resume trainer from checkpoint

**URL:** <https://discuss.huggingface.co/t/cannot-resume-trainer-from-checkpoint/33944>\
**Category:** 🤗Transformers\
**Created:** [March 16, 2023, 1:56pm UTC](https://discuss.huggingface.co/t/cannot-resume-trainer-from-checkpoint/33944 "2023-03-16T13:56:18Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![MultiModal](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@MultiModal](https://discuss.huggingface.co/u/MultiModal)\
**Post date:** [March 16, 2023, 1:56pm UTC](https://discuss.huggingface.co/t/cannot-resume-trainer-from-checkpoint/33944/1 "2023-03-16T13:56:19Z")

</div>

I am trying to resume a training session from a checkpoint. I load the original model and then I call the train(“path/to/checkpoint”) method with a path to the checkpoint. However this refuses to resume from checkpoint. The model was trained with FSDP. could this be affecting nature of the checkpoint saves? This is the error I am getting below.

Loading model from output/checkpoint-18000.  
Traceback (most recent call last):  
File “/home/ubuntu/alpaca/stanford\_alpaca/train.py”, line 246, in   
train()  
File “/home/ubuntu/alpaca/stanford\_alpaca/train.py”, line 239, in train  
trainer.train(“output/checkpoint-18000”)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/transformers/trainer.py”, line 1617, in train  
self.\_load\_from\_checkpoint(resume\_from\_checkpoint)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/transformers/trainer.py”, line 2120, in \_load\_from\_checkpoint  
load\_result = load\_sharded\_checkpoint(model, resume\_from\_checkpoint, strict=is\_sagemaker\_mp\_enabled())  
File “/home/ubuntu/.local/lib/python3.10/site-packages/transformers/modeling\_utils.py”, line 385, in load\_sharded\_checkpoint  
state\_dict = torch.load(os.path.join(folder, shard\_file), map\_location=“cpu”)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/serialization.py”, line 809, in load  
return \_load(opened\_zipfile, map\_location, pickle\_module, \*\*pickle\_load\_args)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/serialization.py”, line 1172, in \_load  
result = unpickler.load()  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/\_utils.py”, line 169, in \_rebuild\_tensor\_v2  
tensor = \_rebuild\_tensor(storage, storage\_offset, size, stride)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/\_utils.py”, line 148, in _rebuild\_tensor  
return t.set_(storage.\_untyped\_storage, storage\_offset, size, stride)  
RuntimeError: Trying to resize storage that is not resizable  
Loading model from output/checkpoint-18000.  
Loading model from output/checkpoint-18000.  
Loading model from output/checkpoint-18000.  
Traceback (most recent call last):  
File “/home/ubuntu/alpaca/stanford\_alpaca/train.py”, line 246, in   
train()  
File “/home/ubuntu/alpaca/stanford\_alpaca/train.py”, line 239, in train  
trainer.train(“output/checkpoint-18000”)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/transformers/trainer.py”, line 1617, in train  
self.\_load\_from\_checkpoint(resume\_from\_checkpoint)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/transformers/trainer.py”, line 2120, in \_load\_from\_checkpoint  
load\_result = load\_sharded\_checkpoint(model, resume\_from\_checkpoint, strict=is\_sagemaker\_mp\_enabled())  
File “/home/ubuntu/.local/lib/python3.10/site-packages/transformers/modeling\_utils.py”, line 385, in load\_sharded\_checkpoint  
state\_dict = torch.load(os.path.join(folder, shard\_file), map\_location=“cpu”)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/serialization.py”, line 809, in load  
return \_load(opened\_zipfile, map\_location, pickle\_module, \*\*pickle\_load\_args)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/serialization.py”, line 1172, in \_load  
result = unpickler.load()  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/\_utils.py”, line 169, in \_rebuild\_tensor\_v2  
tensor = \_rebuild\_tensor(storage, storage\_offset, size, stride)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/\_utils.py”, line 148, in _rebuild\_tensor  
return t.set_(storage.\_untyped\_storage, storage\_offset, size, stride)  
RuntimeError: Trying to resize storage that is not resizable  
Traceback (most recent call last):  
File “/home/ubuntu/alpaca/stanford\_alpaca/train.py”, line 246, in   
train()  
File “/home/ubuntu/alpaca/stanford\_alpaca/train.py”, line 239, in train  
trainer.train(“output/checkpoint-18000”)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/transformers/trainer.py”, line 1617, in train  
self.\_load\_from\_checkpoint(resume\_from\_checkpoint)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/transformers/trainer.py”, line 2120, in \_load\_from\_checkpoint  
load\_result = load\_sharded\_checkpoint(model, resume\_from\_checkpoint, strict=is\_sagemaker\_mp\_enabled())  
File “/home/ubuntu/.local/lib/python3.10/site-packages/transformers/modeling\_utils.py”, line 385, in load\_sharded\_checkpoint  
state\_dict = torch.load(os.path.join(folder, shard\_file), map\_location=“cpu”)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/serialization.py”, line 809, in load  
return \_load(opened\_zipfile, map\_location, pickle\_module, \*\*pickle\_load\_args)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/serialization.py”, line 1172, in \_load  
result = unpickler.load()  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/\_utils.py”, line 169, in \_rebuild\_tensor\_v2  
tensor = \_rebuild\_tensor(storage, storage\_offset, size, stride)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/\_utils.py”, line 148, in _rebuild\_tensor  
return t.set_(storage.\_untyped\_storage, storage\_offset, size, stride)  
RuntimeError: Trying to resize storage that is not resizable  
Traceback (most recent call last):  
File “/home/ubuntu/alpaca/stanford\_alpaca/train.py”, line 246, in   
train()  
File “/home/ubuntu/alpaca/stanford\_alpaca/train.py”, line 239, in train  
trainer.train(“output/checkpoint-18000”)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/transformers/trainer.py”, line 1617, in train  
self.\_load\_from\_checkpoint(resume\_from\_checkpoint)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/transformers/trainer.py”, line 2120, in \_load\_from\_checkpoint  
load\_result = load\_sharded\_checkpoint(model, resume\_from\_checkpoint, strict=is\_sagemaker\_mp\_enabled())  
File “/home/ubuntu/.local/lib/python3.10/site-packages/transformers/modeling\_utils.py”, line 385, in load\_sharded\_checkpoint  
state\_dict = torch.load(os.path.join(folder, shard\_file), map\_location=“cpu”)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/serialization.py”, line 809, in load  
return \_load(opened\_zipfile, map\_location, pickle\_module, \*\*pickle\_load\_args)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/serialization.py”, line 1172, in \_load  
result = unpickler.load()  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/\_utils.py”, line 169, in \_rebuild\_tensor\_v2  
tensor = \_rebuild\_tensor(storage, storage\_offset, size, stride)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/\_utils.py”, line 148, in _rebuild\_tensor  
return t.set_(storage.\_untyped\_storage, storage\_offset, size, stride)  
RuntimeError: Trying to resize storage that is not resizable  
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 123046 closing signal SIGTERM  
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 123048 closing signal SIGTERM  
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local\_rank: 1 (pid: 123047) of binary: /usr/local/bin/python3.10  
Traceback (most recent call last):  
File “/home/ubuntu/.local/bin/torchrun”, line 8, in   
sys.exit(main())  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/ **init**.py”, line 346, in wrapper  
return f(\*args, \*\*kwargs)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/distributed/run.py”, line 794, in main  
run(args)  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/distributed/run.py”, line 785, in run  
elastic\_launch(  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py”, line 134, in **call**  
return launch\_agent(self.\_config, self.\_entrypoint, list(args))  
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py”, line 250, in launch\_agent  
raise ChildFailedError(  
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

---

<div class="post-metadata">

**Author:** ![shiyemin2](https://avatars.discourse-cdn.com/v4/letter/s/a587f6/32.png) [@shiyemin2](https://discuss.huggingface.co/u/shiyemin2)\
**Post date:** [April 9, 2023, 5:28am UTC](https://discuss.huggingface.co/t/cannot-resume-trainer-from-checkpoint/33944/2 "2023-04-09T05:28:45Z")

</div>

Same error. Any solutions?

---

<div class="post-metadata">

**Author:** ![trieudemo11](https://avatars.discourse-cdn.com/v4/letter/t/3ab097/32.png) [@trieudemo11](https://discuss.huggingface.co/u/trieudemo11)\
**Post date:** [May 5, 2023, 9:20am UTC](https://discuss.huggingface.co/t/cannot-resume-trainer-from-checkpoint/33944/3 "2023-05-05T09:20:46Z")

</div>

I’m having the same errors when loading a checkpoint using torch.load(checkpoint)