# Exact difference between Transformers' and Accelerate's DeepSpeed integrations?

**URL:** <https://discuss.huggingface.co/t/exact-difference-between-transformers-and-accelerates-deepspeed-integrations/66843>\
**Category:** DeepSpeed\
**Created:** [December 23, 2023, 9:17pm UTC](https://discuss.huggingface.co/t/exact-difference-between-transformers-and-accelerates-deepspeed-integrations/66843 "2023-12-23T21:17:46Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jkkim](https://avatars.discourse-cdn.com/v4/letter/j/46a35a/32.png) [@jkkim](https://discuss.huggingface.co/u/jkkim)\
**Post date:** [December 23, 2023, 9:17pm UTC](https://discuss.huggingface.co/t/exact-difference-between-transformers-and-accelerates-deepspeed-integrations/66843/1 "2023-12-23T21:17:46Z")

</div>

Hello,

I’m trying to use DeepSpeed with Transformers, and I see there are two DeepSpeed integrations documented on HF:  
(a) Transformers’ DeepSpeed integration: [DeepSpeed Integration](https://huggingface.co/docs/transformers/main/main_classes/deepspeed)  
(b) Accelerate’s DeepSpeed integration: [DeepSpeed](https://huggingface.co/docs/accelerate/usage_guides/deepspeed)

However, I’m a bit confused by these two.  
They have separate documentations, but are they really two completely separate integrations?

After examining the codes, I’ve realized that `Accelerator._prepare_deepspeed` calls `deepspeed.initialize` while there is no calling of `deepspeed.initialize` on Transformers’ end, which seems to be contradicting with documentation (a) as “Trainer Deepspeed Integration” shouldn’t require the user to call `deepspeed.initialize` manually.

Could someone clarify it for me, please?

Thank you!

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [December 23, 2023, 9:34pm UTC](https://discuss.huggingface.co/t/exact-difference-between-transformers-and-accelerates-deepspeed-integrations/66843/2 "2023-12-23T21:34:02Z")

</div>

The Trainer uses Accelerate under the hood so it’s called when the Trainer calls `accelerator.prepare()`

---

<div class="post-metadata">

**Author:** ![jkkim](https://avatars.discourse-cdn.com/v4/letter/j/46a35a/32.png) [@jkkim](https://discuss.huggingface.co/u/jkkim)\
**Post date:** [December 23, 2023, 9:45pm UTC](https://discuss.huggingface.co/t/exact-difference-between-transformers-and-accelerates-deepspeed-integrations/66843/3 "2023-12-23T21:45:07Z")

</div>

Thank you for your response.

Does that mean Transformers’ DeepSpeed integration relies on Accelerate’s DeepSpeed integration?

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [December 23, 2023, 9:46pm UTC](https://discuss.huggingface.co/t/exact-difference-between-transformers-and-accelerates-deepspeed-integrations/66843/4 "2023-12-23T21:46:56Z")

</div>

Yes indeed

---

<div class="post-metadata">

**Author:** ![SantoshScienceIO](https://avatars.discourse-cdn.com/v4/letter/s/ac8455/32.png) [@SantoshScienceIO](https://discuss.huggingface.co/u/SantoshScienceIO)\
**Post date:** [December 25, 2023, 1:59am UTC](https://discuss.huggingface.co/t/exact-difference-between-transformers-and-accelerates-deepspeed-integrations/66843/5 "2023-12-25T01:59:22Z")

</div>

so in an example where the script includes

```auto
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

training_args = TrainingArguments(
    ...
    deepspeed='ds_config.json'
)

trainer = Trainer(
        model=model,
        tokenizer=tokenizer,
        args=training_args
)
trainer.train()

```

We would use

`deepspeed train.py`

instead of

`accelerate launch --config_file config.yaml train.py`

?

Or would those two be equivalent?

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [February 13, 2024, 10:43pm UTC](https://discuss.huggingface.co/t/exact-difference-between-transformers-and-accelerates-deepspeed-integrations/66843/6 "2024-02-13T22:43:17Z")

</div>

Equivalent