# Way to train a basic Transformer

**URL:** <https://discuss.huggingface.co/t/way-to-train-a-basic-transformer/2145>\
**Category:** Beginners\
**Created:** [November 20, 2020, 10:11am UTC](https://discuss.huggingface.co/t/way-to-train-a-basic-transformer/2145 "2020-11-20T10:11:00Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![asvs](https://avatars.discourse-cdn.com/v4/letter/a/cdc98d/32.png) [@asvs](https://discuss.huggingface.co/u/asvs)\
**Post date:** [November 20, 2020, 10:11am UTC](https://discuss.huggingface.co/t/way-to-train-a-basic-transformer/2145/1 "2020-11-20T10:11:00Z")

</div>

Is there any way to train a basic Transformer (the transformer architecture presented in the original Attention Is All You Need paper). I’m trying to set up a translation baseline on my custom dataset (previously trained models do not exist for the language). If anyone can point out a ready-made implementation of just the basic Transformer it would be very helpful.

Thanks!

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [November 20, 2020, 12:04pm UTC](https://discuss.huggingface.co/t/way-to-train-a-basic-transformer/2145/2 "2020-11-20T12:04:43Z")

</div>

I think OpenNMT is the better framework to do something like that. It allows to train a model that you describe easily. `transformers` is more directed towards specialized, specific transformer architectures.

---

<div class="post-metadata">

**Author:** ![asvs](https://avatars.discourse-cdn.com/v4/letter/a/cdc98d/32.png) [@asvs](https://discuss.huggingface.co/u/asvs)\
**Post date:** [November 20, 2020, 2:59pm UTC](https://discuss.huggingface.co/t/way-to-train-a-basic-transformer/2145/3 "2020-11-20T14:59:18Z")

</div>

Thanks! I used `fairseq` a few times. I think OpenNMT is kinda similar, they seem more black box like not full control on the entire pipeline. That’s why I wanted to know if such a thing is possible with transformers because this is native Python and not command-line tools.

---

<div class="post-metadata">

**Author:** ![abhishek](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/abhishek/32/32360_2.png) [@abhishek](https://discuss.huggingface.co/u/abhishek)\
**Post date:** [November 20, 2020, 10:36pm UTC](https://discuss.huggingface.co/t/way-to-train-a-basic-transformer/2145/4 "2020-11-20T22:36:29Z")

</div>

Have you seen this: [https://pytorch.org/tutorials/beginner/transformer\_tutorial.html](https://pytorch.org/tutorials/beginner/transformer_tutorial.html) ?

---

<div class="post-metadata">

**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](https://discuss.huggingface.co/u/Jung)\
**Post date:** [November 21, 2020, 3:06am UTC](https://discuss.huggingface.co/t/way-to-train-a-basic-transformer/2145/5 "2020-11-21T03:06:22Z")

</div>

And this is TF2-Keras equivalent to @abhishek’s recommendation 😃

> **[Neural machine translation with a Transformer and Keras  |  Text...](https://www.tensorflow.org/text/tutorials/transformer)**

---

<div class="post-metadata">

**Author:** ![asvs](https://avatars.discourse-cdn.com/v4/letter/a/cdc98d/32.png) [@asvs](https://discuss.huggingface.co/u/asvs)\
**Post date:** [November 21, 2020, 6:20am UTC](https://discuss.huggingface.co/t/way-to-train-a-basic-transformer/2145/6 "2020-11-21T06:20:43Z")

</div>

Thank you! I implemented mine from scratch in Pytorch and Lightning with the help of [bentrevett’s notebook](https://github.com/bentrevett/pytorch-seq2seq/blob/master/6%20-%20Attention%20is%20All%20You%20Need.ipynb) and [source code of BART](https://huggingface.co/transformers/_modules/transformers/configuration_bart.html#BartConfig) in the transformers library.

---

<div class="post-metadata">

**Author:** ![asvs](https://avatars.discourse-cdn.com/v4/letter/a/cdc98d/32.png) [@asvs](https://discuss.huggingface.co/u/asvs)\
**Post date:** [November 21, 2020, 6:21am UTC](https://discuss.huggingface.co/t/way-to-train-a-basic-transformer/2145/7 "2020-11-21T06:21:16Z")

</div>

Thank you for the resources.