# Load fsdp+lora checkpoint error

**URL:** <https://discuss.huggingface.co/t/load-fsdp-lora-checkpoint-error/96602>\
**Category:** Beginners\
**Created:** [July 10, 2024, 1:31pm UTC](https://discuss.huggingface.co/t/load-fsdp-lora-checkpoint-error/96602 "2024-07-10T13:31:22Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![0aaAddD0](https://avatars.discourse-cdn.com/v4/letter/0/f04885/32.png) [@0aaAddD0](https://discuss.huggingface.co/u/0aaAddD0)\
**Post date:** [July 10, 2024, 1:31pm UTC](https://discuss.huggingface.co/t/load-fsdp-lora-checkpoint-error/96602/1 "2024-07-10T13:31:22Z")

</div>

File “/home/wuzh/zd/GIT-Mol/crystal-text-llm-main/llama\_finetune.py”, line 522, in   
main(args)  
File “/home/wuzh/zd/GIT-Mol/crystal-text-llm-main/llama\_finetune.py”, line 481, in main  
train\_result = trainer.train(resume\_from\_checkpoint=args.resume\_dir)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/transformers/trainer.py”, line 1932, in train  
return inner\_training\_loop(  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/transformers/trainer.py”, line 2268, in \_inner\_training\_loop  
tr\_loss\_step = self.training\_step(model, inputs)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/transformers/trainer.py”, line 3307, in training\_step  
loss = self.compute\_loss(model, inputs)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/transformers/trainer.py”, line 3338, in compute\_loss  
outputs = model(\*\*inputs)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1518, in \_wrapped\_call\_impl  
return self.\_call\_impl(\*args, \*\*kwargs)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1527, in \_call\_impl  
return forward\_call(\*args, \*\*kwargs)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/accelerate/utils/operations.py”, line 819, in forward  
return model\_forward(\*args, \*\*kwargs)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/accelerate/utils/operations.py”, line 807, in **call**  
return convert\_to\_fp32(self.model\_forward(\*args, \*\*kwargs))  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/torch/amp/autocast\_mode.py”, line 16, in decorate\_autocast  
return func(\*args, \*\*kwargs)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/torch/\_dynamo/eval\_frame.py”, line 328, in \_fn  
return fn(\*args, \*\*kwargs)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/torch/\_dynamo/external\_utils.py”, line 17, in inner  
return fn(\*args, \*\*kwargs)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1518, in \_wrapped\_call\_impl  
return self.\_call\_impl(\*args, \*\*kwargs)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1527, in \_call\_impl  
return forward\_call(\*args, \*\*kwargs)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/torch/distributed/fsdp/fully\_sharded\_data\_parallel.py”, line 823, in forward  
args, kwargs = \_root\_pre\_forward(self, self, args, kwargs)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/torch/distributed/fsdp/\_runtime\_utils.py”, line 558, in \_root\_pre\_forward  
\_lazy\_init(state, module)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/torch/distributed/fsdp/\_runtime\_utils.py”, line 173, in \_lazy\_init  
\_share\_state\_and\_init\_handle\_attrs(state, root\_module)  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/torch/distributed/fsdp/\_runtime\_utils.py”, line 261, in \_share\_state\_and\_init\_handle\_attrs  
\_p\_assert(  
File “/work/zd/anaconda/fsdp/lib/python3.9/site-packages/torch/distributed/utils.py”, line 145, in \_p\_assert  
traceback.print\_stack()  
Non-root FSDP instance’s `_is_root` should not have been set yet or should have been set to `False`

when using the “train\_result = trainer.train(resume\_from\_checkpoint=args.resume\_dir)” to load the checkpoint.  
here is the library code:  
"\_p\_assert(  
fsdp\_state.\_is\_root is None or not fsdp\_state.\_is\_root,  
“Non-root FSDP instance’s `_is_root` should not have been "  
“set yet or should have been set to `False`”,  
)  
fsdp\_state.\_is\_root = False  
fsdp\_state.\_unshard\_stream = root\_state.\_unshard\_stream”