# \[Solved\] Cannot restart training from deepspeed checkpoint

**URL:** <https://discuss.huggingface.co/t/solved-cannot-restart-training-from-deepspeed-checkpoint/19786>\
**Category:** Intermediate\
**Created:** [June 29, 2022, 2:08pm UTC](https://discuss.huggingface.co/t/solved-cannot-restart-training-from-deepspeed-checkpoint/19786 "2022-06-29T14:08:22Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ierezell](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ierezell/32/2517_2.png) [@ierezell](https://discuss.huggingface.co/u/ierezell)\
**Post date:** [June 29, 2022, 2:08pm UTC](https://discuss.huggingface.co/t/solved-cannot-restart-training-from-deepspeed-checkpoint/19786/1 "2022-06-29T14:08:22Z")

</div>

Hello 🙂

I’m using deepspeed to train big models on two GPUs (that cannot fit on one).  
I’m using the hugging face Trainer with deepspeed params like so :

```python

trainer = Trainer(
    model=model,
    train_dataset=CustomDataset(...),
    eval_dataset=CustomDataset(...),
    tokenizer=tokenizer,
    compute_metrics=custom_metric,
    data_collator=DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False),
    args=TrainingArguments(
        output_dir=f"output_{backbone.replace('/', '_')}",
        do_train=True,
        do_eval=True,
        do_predict=True,
        evaluation_strategy="steps",
        per_device_train_batch_size=2,
        per_device_eval_batch_size=2,
        num_train_epochs=5,
        fp16=True,
        deepspeed={
            "fp16": {
                "enabled": True,
                "min_loss_scale": 1,
                "opt_level": "O3",
            },
            "wandb": {
                "enabled": True,
                "team": "my_team",
                "project": "my_project"
            },
            "optimizer": {
                "type": "AdamW",
                "params": {
                    "lr": "auto",
                    "betas": "auto",
                    "eps": "auto",
                    "weight_decay": "auto",
                },
            },
            "scheduler": {
                "type": "WarmupLR",
                "params": {
                    "warmup_min_lr": "auto",
                    "warmup_max_lr": "auto",
                    "warmup_num_steps": "auto",
                },
            },
            "zero_optimization": {
                "stage": 3,
                "offload_optimizer": {"device": "cpu"},
                "offload_param": {"device": "cpu"},
                "allgather_partitions": True,
                "allgather_bucket_size": 5e8,
                "contiguous_gradients": True,
            },
            "train_batch_size": 4,
            "train_micro_batch_size_per_gpu": 2,
        },
        report_to="wandb",
    ),
)

```

The first training went fine and I have folders checkpoint with :

```auto
global_step29500
|--zero_pp_rank_0_mp_rank_00_model_states.pt
|--zero_pp_rank_0_mp_rank_00_optim_states.pt
special_tokens_map.json
trainer_state.json
zero_to_fp32.py
config.json
merges.txt
tokenizer.json
training_args.bin
latest       
rng_state_0.pth
tokenizer_config.json    
vocab.json

```

However when I try to `resume_from_checkpoint` (in `trainer.train()`) it says :  
`raise ValueError(f"Can't find a valid checkpoint")`

So I try to use the `zero_to_fp32.py` file to create a latest.ckpt but without success.

The doc of deepspeed says something about “automatically loading weights” but I’m sceptic.

Thanks in advance,  
Have a great day.

---

<div class="post-metadata">

**Author:** ![ierezell](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ierezell/32/2517_2.png) [@ierezell](https://discuss.huggingface.co/u/ierezell)\
**Post date:** [June 29, 2022, 4:34pm UTC](https://discuss.huggingface.co/t/solved-cannot-restart-training-from-deepspeed-checkpoint/19786/2 "2022-06-29T16:34:02Z")

</div>

For the posterity :

After reading the huggingface code that loads models, you have to use `zero_to_fp32.py` but absolutely having the file named `"pytorch_model.bin"` (dummy mistake from my side… 😓).

It still can load even without the `pytorch_model.bin.index.json`

Hope this will help someone someday 🙂

---

<div class="post-metadata">

**Author:** ![lucasjin](https://avatars.discourse-cdn.com/v4/letter/l/f4b2a3/32.png) [@lucasjin](https://discuss.huggingface.co/u/lucasjin)\
**Post date:** [May 30, 2023, 12:36pm UTC](https://discuss.huggingface.co/t/solved-cannot-restart-training-from-deepspeed-checkpoint/19786/3 "2023-05-30T12:36:27Z")

</div>

I met same problem.

Can u elebarate more?

I already have pytorch\_model.bin but still can not load

---

<div class="post-metadata">

**Author:** ![HereWe](https://avatars.discourse-cdn.com/v4/letter/h/f1d935/32.png) [@HereWe](https://discuss.huggingface.co/u/HereWe)\
**Post date:** [December 28, 2023, 3:41am UTC](https://discuss.huggingface.co/t/solved-cannot-restart-training-from-deepspeed-checkpoint/19786/4 "2023-12-28T03:41:09Z")

</div>

It works for me!