# Implementing a custom Attention Transformer **URL:** \ **Category:** Awesome paper\ **Created:** [September 3, 2021, 4:54am UTC](https://discuss.huggingface.co/t/implementing-a-custom-attention-transformer/9702 "2021-09-03T04:54:27Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![iakarshu](https://avatars.discourse-cdn.com/v4/letter/i/a3d4f5/32.png) [@iakarshu](https://discuss.huggingface.co/u/iakarshu)\ **Post date:** [September 3, 2021, 4:54am UTC](https://discuss.huggingface.co/t/implementing-a-custom-attention-transformer/9702/1 "2021-09-03T04:54:27Z")
Hello everyone, currently I am trying to implement a custom attention transformer, whose attention is given on Page No. 4 of this [link](https://arxiv.org/pdf/2106.11539.pdf). They have used hugging face for the implementation, and I am not sure about how to go for approaching this problem, and how to use hugging face to implement custom attention. Can anybody guide me, about how to go about implementing this? Thanks, ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [September 3, 2021, 8:24pm UTC](https://discuss.huggingface.co/t/implementing-a-custom-attention-transformer/9702/2 "2021-09-03T20:24:28Z")
Hey @iakarshu my best guess is that the authors implemented DocFormer from scratch, so as far as I can tell you can’t do some clever subclassing of an existing model to tweak the attention layers. Having said that, you could look at the implementation of [LayoutLMV2](https://github.com/huggingface/transformers/tree/master/src/transformers/models/layoutlmv2) which seems to share a similar approach and you can use [this template](https://github.com/huggingface/transformers/tree/master/templates/adding_a_new_model) to get all the basic modeling files. Do you know if AWS open-sourced the pretrained weights of DocFormer? Without them, you might need a lot of compute to build a useful model. Hope that helps! ---
**Author:** ![iakarshu](https://avatars.discourse-cdn.com/v4/letter/i/a3d4f5/32.png) [@iakarshu](https://discuss.huggingface.co/u/iakarshu)\ **Post date:** [September 4, 2021, 3:02am UTC](https://discuss.huggingface.co/t/implementing-a-custom-attention-transformer/9702/3 "2021-09-04T03:02:44Z")
Hey @lewtun, thanks a lot for sharing this, maybe then I would focus on implementing it from scratch, and learn from the implementation of LayoutLMV2, thanks a lot for that. And for the computation, I have some resources, which means NVIDIA DGX to work, and I am searching about the open-source Docformer code, but I am not getting it. I mailed the author and they refrained from sharing the code, so I don’t think that they have open-sourced it. Again, thanks a lot for replying. ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [September 6, 2021, 8:35am UTC](https://discuss.huggingface.co/t/implementing-a-custom-attention-transformer/9702/4 "2021-09-06T08:35:30Z")
Hey @nielsr is [DocFormer](https://arxiv.org/abs/2106.11539) currently on your roadmap for `transformers`? @iakarshu is thinking about having a go at implementing and pretraining it (because the authors didn’t release code or weights), so I thought it would be good to double-check that you don’t do the same work twice 🙂 ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [September 6, 2021, 9:48am UTC](https://discuss.huggingface.co/t/implementing-a-custom-attention-transformer/9702/5 "2021-09-06T09:48:59Z")
No it’s not on my list, seems interesting. However, if there are no pre-trained weights available (and even no code), then there’s a low chance for me to add it to the library. ---
**Author:** ![iakarshu](https://avatars.discourse-cdn.com/v4/letter/i/a3d4f5/32.png) [@iakarshu](https://discuss.huggingface.co/u/iakarshu)\ **Post date:** [September 6, 2021, 10:14am UTC](https://discuss.huggingface.co/t/implementing-a-custom-attention-transformer/9702/6 "2021-09-06T10:14:45Z")
@nielsr @lewtun thanks a lot, then I would do it, and would ask the community if i get stucked, thanks a lot, I shall begin my coding then 🙂